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ABSTRACT 

This notebook contains 21 standards established by 
the Center for Educational Statistics (CES); CES legislation; and a 
description of the CES publication and publication review processes. 
The publication clearance procedures and the external advice and peer 
review policies are established by the Office of Educational Research 
and Improvement (OERI) of which the CES is a part. The standards 
described here cover all phases of the Center's work, including (1) 
planning; (2) Request for Proposals (RFP) development and contract 
monitoring; (3) implementation, data collection and processing; (4) 
release and publication of data; and (5) sampling and non-sampling 
error. Each standard has a three-field reference number, encoding the 
year the standard was implemented, the subject group of the standard, 
and its sequence within the subject group. These standards are 
intended as guidelines for CES staff and contractors in their data 
collection, analysis, and dissemenation, and for users of the data in 
making appropriate applications of CES information. Appendices 
include the General Education Provisions Act, Sections 406 (mandating 
the Center for Education Statistics) and 405 (mandating the National 
Assessment of Educational Progress), the Carl D. Perkins Vocational 
Education Act, Part C, Sections 421, 422, and 423, and a statement of 
OERI's external advice and peer review policy. (JGL) 
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FOREWORD 



I am very pleased to announce the adoption of standards for the Center for 
Education Statistics. This notebook contains the full set of standards, 
adopted as of March 16, 1987. It also contains CES legislation and the 
CES publication review process. 

CES standards must be followed by all Center staff and contractors: 

o Performance agreements for Center staff are to include 
a section on responsibilities to adhere to and administer 
the standards. 

o All Requests for Proposals (RFP f s) must include a section 
referencing the standards and requiring their use in work 
conducted for the Center. 

o New staff to the Center will be introduced to the structure 
and mission of the Center and each of the standards. 

In the fall, the Chief Statistician will initiate a formal evaluation of 
the standards program to ensure that the standards have been implemented 
in all phases of the Center's work, and to review their operational 
feasibility. We will probably find some inconsistencies among the 
standards as we work with them, and these will be reconciled during the 
review and revised. In a year, we expect to reissue the standards. 

Adoption of a set of written standards is important to any Federal 
statistical agency. They codify how we expect to behave professionally, 
and indicate the basis on which we expect to be judged by our peers in the 
statistical community. They represent the minimum level of quality and 
effort we would expect in any of our efforts or those of our contractors 
and grantees. For the Center, they provide a means for and assurance of 
consistency between and within the studies the Center conducts. Finally, 
users of Center data have before them clear documentation of methods and 
principles the Center employs in the collection of data. 

I consider adoption of the standards to be the Center's most visible 
declaration of commitment to the highest quality in all of its products. 



/ 



Emerson J. Elliott 
Director 
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Status 



There are currently 21 standards adopted by the Center, presented in this 
notebook. In the fall of 1987, a review of each of the standards will be 
conducted, with revisions issued in th^ spring of 1988. The review will 
consider whether the standards have been put into operation, what aspects of 
the standards have kept the standards from being fully implemented, whether 
there remain any gaps in the standards, and whether the standards are 
consistent among themselves. The standards will also be reviewed with an eye 
towards whether they are clear in setting forth attainable goals or present 
instead prosaic but unattainable ideals. 

Contents 

This notebook contains eight sections: 
o Standards on Planning 

o Standards on RFP Developraent/Contract Monitoring 

o Standards on Project Implementation/Data Collection/Processing 

o Standards on Release/Publication of Data 

o Standards on Sampling and Non-Sampling Error 

o CES Legislation 

o Publication Policy 

o Other 

Finally, note that each standard has a three-field reference number. The first 
field refers to the year the standard was implemented, the second field refers 
to the subject group of the standard, and the last field is the sequence number 
of the standard within its subject group. These numbers have been added to 
facilitate cross references among the standards. 
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SUBJECT: STANDARD FOR PLANNING STATISTICAL SURVEYS 

EFFECTIVE DATE: 12/08/86 CES STANDARD 86.01-01 

PURPOSE: To ensure that the surveys conducted by CES are thoroughly planned in 
terms of goals, related surveys and research, methodology, Implementation and 
user needs. 

o CES adopts as its standard for planning 0MB Directive No 1: Standards 
for Statistical Surveys 

o The survey must be justified in terms of information needs, hypotheses 
to be tested or problems to be solved. 

o Related studies, surveys and reports of Federal and non-Federal sources 
must be reviewed to ensure that part or all the data are not available 
from some existing source or could not be more appropriately obtained by 
adding questions to an existing survey sponsored by CES or other 
agencies . 

o Representatives of institutions or groups from whom .^formation will be 
collected and/or which will use the data must be consulted. 

o A study plan must be developed that analyzes target population and 
extent of coverage; sampling; frequency and timing; method of 
collection; considerations of sampling and nonsampling errors; 
evaluation of survey data; allowance for pretests and pilot tests; 
provisions for follow-up; time lines and milestones; and cost estimates. 

o A preliminary analysis plan must be developed that Identifies analysis 
issues, major variables and proposed statistical techniques. 

o A preliminary publication and dissemination plan identifying primary 
users and major publications tailored for these users must be developed. 



RELATED STANDARDS: 

o 0MB Directive No 1: Standards for Statistical Surveys, Statistical 
Policy Handbook , Federal Statistical Policy and Standards, U.S. 
Department of Commerce, 1978 
o 0ERI Peer Review Policy 

RELATED CHECKLISTS, FORMS AND DOCUMENTS: 

o Planning Statistical Surveys Checklist 
o Standard Form 83 (0MB Clearance) 

o Statistical Policy forking Pa per 9: Contracting for Surveys . Office of 
Management and Budget, 1983 

- 1 - 
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CHECKLIST 



PLANNING STATISTICAL SURVEYS 



A. Purpose of the Survey 

1. The survey is justified in terms of information needs, hypotheses to 
be tested or problems to be solved. 

2. The specific purposes are consistent with appropriate legislative 
mandates and agency priorities. 

3. Research, policy and/or programmatic questions were used to guide the 
development of the survey. 

B. Related Studies and Surveys 

1. Existing information or related surveys were reviewed to ensure that a 
new data collection effort is needed. 

2. Research and analysis reports on similar topics were reviewed during 
the design of the study. 

3. Problems identified in previous or related surveys are corrected or 
documented • 

C. Outside Consultation 

1. Representatives of institutions or groups from who/j data will be 
collected and potential users of the data were consulted during the 
planning process. 

2. OERI Peer Review Policy was followed. 



D. Survey Plan 

1. The target population and extent of coverage are identified and are 
appropriate for the goals of the survey. 

2. At least a preliminary sample design is developed including 

a. definition of the sampling frame at each stage of sample selection 

b. criteria or definitions of strata at each stage 

c. strategy for sample selection at each stage (i.e., stratification, 
etc.) 

d. levels of preci * ^n for critical variables 

3. The proposed frequency and timing of the survey are justified in terms 
of the goals of the survey and the need to limit data collection 
burden. 

- 2 - 
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CENTER FOR EDUCATION STATISTICS 
OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT 
U. S. DEPARTMENT OF EDUCATION 



SUBJECT: STANDARD FOR INDEPENDENT EVALUATION OF SURVEYS 
EFFECTIVE DATE: 03/16/87 



CES STANDARD 87-01-02 



PURPOSE: To ensure that the data collected are of v.he highest quality 
consistent with cost and other constraints, and that corroborating evidence can 
be found regarding the information collected in CES surveys. 

o At the time a survey is being planned and developed, a research plan 
should also be developed that will allow the independent evaluation of 
the survey. This plan should identify possible sources of nonsampling 
error in the survey and indicate how these will be dealt with and 
measured. In addition, the plan should list how data will be obtained 
for the evaluation of the survey: through checks for internal 
consistency, external comparison against other data sources, and 
experiments built into the survey to study alternative data collection 
methodologies . 

o Where possible, the pretest for a survey should include some experiments 
designed to study sources and control of nonsampling error. Normally, 
no survey should be initiated without a formal pretest and evaluation of 
the pretest results. 

o Items should be identified that can be used in the internal checking 
process once the data has been collected in a survey. Tables should be 
generated to study this data before any other set of data from the 
survey, and results of this analysis should be used to correct the data 
if outliers are found or data are found to be inconsistent. 

o Sources outside the data collected in a survey should be identified and 
put to use to analyze the survey results. Again, tables should be 
generated and compared to these outside sources to help ensure the 
validity of th£ data before analysis of the data begins. 

o The study plan for the survey or data ct/llection operation should have 
an ongoing research plan built: into the study design. This moans that 
consideration should bo given to having a small lndopendont replicate of 
tho sample proldentlf led for use as a test group while the survey 
continues. This is especially important for longitudinal studies, where 
design changes, new items, or other effects introduced to the survey 
without prior testing can severely affect the time series. 



RELATED STANDARDS : 

o CES Standard for Planning Statistical Surveys 86-01-01 

o CES Standard for Testing Data Collection Instruments 87-03-01 

o CES Standard for Technical Documentation in Data Releases 87-04-02 

RELATED CHECKLISTS AND FORMS: 



- 4 - 
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SUBJECT: STANDARD FOR RFP DEVELOPMENT FOR SURVEYS 

EFFECTIVE DATE: 02/09/87 CES STANDARD 87-02-01 

PURPOSE: To ensure that RFPs for survey research Issued by t:he Center for 
Education Statistics contain the information necessary to enable all 
prospective offerors to be equally responsive in submitting a proposal, and to 
ensure that the proposals submit d address the needs of CES, 

o RFPs issued by CES for survey research should specify a technically 
sound survey design that is consistent with project objectives. For 
sample surveys, the statement of work should discuss the survey design 
including the sampling frame or method of obtaining the universe, method 
of selecting the sample at each stage, expected number of sampling 
units, and the sample size or required level of precision for estimates 
for key variables; data collection method; either a draft questionnaire 
ov specific instructions on objectives, length of interview and data 
specifications; requirements for pretesting; ADP processing 
requirements; and weighting procedures. Under some circumstam s, an RFP 
can require the offerors to propose a research design consistent with 
the survey objectives. 

o The Project Officer has primary responsibility for writing the scope of 
work of the RFP. The Project Officer must arrange for technical input 
prior to writing the scope of work from CES staff and, in accordance 
with OERI Peer Review Policy, outside experts from the Department of 
Education, other Federal agencies and universities, or the private 
sector. Input should occur at several points during the development of 
the survey, but is essential before final decisions are made on the 
survey design, and before final RFP preparation, 

o Evaluation criteria must include points for technical quality 

(understanding of the issues, sampling, interviewer training, field 
work, editing and coding); appropriateness and feasibility of proposed 
milestones; experience of the offeror, experience of key staff, and 
management plan. The distribution of points will depend on the nature 
of the procurement. 



RELATED STANDARDS: 

o CES Standard for Planning Statistical Surveys, 86-01-01 

o CES Standard for Monitoring Survey Contracts, 87-02-02 

o CES Standard for Maintaining Contract Files, 87-02-03 

o CES Standard for Codes, Abbreviations and Acronyms, 87-03-05 

o CES Standard for Survey Documentation, 87-04-01 

o OERI Peer Review Policy 

RELATED CHECKLISTS, FORMS AND DOCUMENTS: 

° Statistical Policy Workin g Paper 9 : Contracting for Surveys f Office of 
Management and Budget, June, 1983, 

- 5 - 
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SUBJECT: STANDARD FOR MONITORING SURVEY CONTRACTS 

EFFECTIVE DATE: 02/09/37 CES STANDARD 87-02-02 

PURPOSE: To ensure that cvAtracts for CES surveys are performed as written, 
Project Officers have the following responsibilities. 

o ADMINISTRATIVE RESPONSIBILITIES: 

- Maintain a current contract file. 

Provide technical direction to contractors, as necessary and 
appropriate, depending on the type and terms of the contract. 

Review and make recommendations to the Contracting Officer as to the 
approval, disapproval, or other action to take concerning a 
contractor's submission of (or failure to submit) payment request 3, 
deliverables, interim or final progress or financial reports, or any 
other requirements of the contract. 

- Notify CES officials of developing problems, in a timely manner. 

- Assure that the contractor has access to or use of government 
property or services as agreed to in the contract. 

- Make recommendations on contract modifications, if necessary, and 
ensure that changes are made only after notification by contracts 
officer. 

- Prepare closeout documentation. 

o QUALITY ASSURANCE RESPONSIBILITIES: 

Hold meetings with the contractor and advisory panels which the 
contractor must attend. 

- Insure that OERI peer review policy and guidelines with reference to 
quality control are followed. 

Monitor critical stages in the development and implementation of the 
survey to ensure that the objectives of the survey are met. Critical 
stages include sample selection, questionnaire development, training 
of data collectors, codir.g of responses and preparation of data 
documentation. Project Officer should also monitor the overall 
response rates and, where appropriate, critical item response rates. 

Provide written reviews of contract deliverables for completion, 
quality and accuracy. Reviews should be completed within two weeks 
of receipt. 

- Prepare summary of project history and results. 
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STANDARD FOR MONITORING SURVEY CONTRACTS 



RELATED STANDARDS: 

o CES Standard for Maintaining Contract Files, 87-02-03 

o CES Standard for Survey Documentation 87-04-01 

o CES Standard for Maintaining Confidentiality, 87-04-03 

o OERI Peer Review Policy 

RELATED CHECKLISTS, FORMS AND DOCUMENTS: 

o Contract Monitoring for Program Officials . Departmental Directive, U.S. 
Education Department, 1987. 

o Statistical Poli cy Working Paper 9: Contracting for Surveys . Office of 
Management and Budget, June, 1983. 

o The Negotiated Co ntracting Process: A Guide for Project Officers . U.S. 
Department of Health, Education and Welfare, 1977, 
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CENTER FOR EDUCATION STATISTICS 
OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT 
U.S. DEPARTMENT OF EDUCATION 



SUBJECT: STANDARD FOR MAINTAINING CONTRACT FILES 

EFFECTIVE DATE: 02/09/87 CES STANDARD 87-02-03 

PURPOSE: To ensure that contract files contain all information about the 
requirements and conduct of a contract. 

o Project Officers are responsible for maintaining an up-to-date file on 
each contract. The file must be maintained for the same period as 
official files (those maintained in the contracting office) normally 6 
years and 3 months after final payment has been made. This file will 
include: 



- Planning documents, notes from planning meetings, and comments from 
peer reviewers. 

- Copies of relevant legislation and Department policies, if applicable. 

- The RFP; PAR, and Government cost estimates. 

- The Recommendation for Award memo and Technical Evaluation Report 
including individual evaluations of panel members . 

- Contract arid modifications. 

- Copies of approved vouchers . 

- Deliverables and copies of written reviews of the deliverables by the 
project officer and other peer reviewers. 

- Copies of signed correspondence between project officer or contract 
officer and the contractor. 



- Dated notes of meetings or substantive telephone conversations between 
project officer and contractor. 

- Dated notes of meetings or substantive telephone conversations with 
project officer and contract officer. 

- Approvals by Assistant Secretary and Director, CES. 

- 0MB approval package and approval notifications. 

- Copies of technical documentation prepared in accordance with CES 
Standard for Survey Documentation. 

- Project Officer's summary and evaluation of the project. 



RELATED STANDARDS: 

o CES Standard for Survey Documentation, 86-v4-01 

o CES Standard for Monitoring Survey Contracts, 87-02-02 

o CES Standard for Maintaining Confidentiality, 87-04-03 

RELATED CHECKLISTS, FORMS AND DOCUMENTS: 

o Comprehensive Records Disposition Schedules, 05-83, U.S. Department of 
Education 

erJc * 5 
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SUBJECT: STANDARD FOR TESTING DATA COLLECTION INSTRUMENTS AND DATA COLLEC r ION 
SYSTEMS 

EFFECTIVE DATE: 02/09/87 CES STANDARD 87-03-01 

PURPOSE: To ensure that the data instruments and systems used by CES 
to collect data are adequately tested before a survey is conducted. 

o The Project Officer must review questionnaires to determine that each 

question is needed for the survey objectives and that overall the questions 
will provide the needed data. 

o Questionnaires developed for all CES surveys must be pretested for 

feasibility before field testing by the Project Officer, other CES staff and 
representatives of persons and groups who will participate in the survey. 
The questionnaire should be reviewed for clarity, flow of questions, adequacy 
of instructions and length of response time, appropriate grouping and 
sequencing of questions. 

o Once a questionnaire is developed it must be pretested in the field. Sample 
size for the pretest should be sufficiently large to identify potential 
problems and therefore should not be limited to fewer than ten to avoid 0MB 
clearance procedures. In most cases, purposive sample designs for a pretest 
are preferable to probability samples in order to ensure that important 
subpopulations are included. 

o Provision for successive pretests should be made when the first indicates the 
need for a substantial change in the questionnaire or the system. 

o The method or system by which the data will be collected must also be tested. 
For face-to-face and pre-arranged telephone interviews this includes 
procedures for contacting respondents, setting up meeting times and places, 
notifying respondents of materials they may need to review or have at the 
interview, and other logistical considerations. For mailed questionnaires, 
procedures for sending and receiving the questionnaire, logging responses and 
follow-through on non-response must be tested. 

o For surveys based on an analysis of administrative records, a field-based 
test of the survey collection instrument must be done to ensure that the 
information requested is available without undue data burden; that 
definitions and instructions are provided and are comparable among the 
reporting units; and that appropriate staff to respond are identified. 

o Quality control procedures should be tested before beginning a major survey. 
Quality control procedures may include validity checking, data-editing, 
followup on non-response and data processing checks. 

RELATED STANDARDS: 

o CES Standard for Planning Statistical Surveys, 86-01-01 
o CES Standard for RFP Development for Surveys, 87-02-01 

RELATED CHECKLISTS, FORMS AND DOCUMENTS 

o Testing Data Collection Instruments and Data Collection Systems 
Checklist. 



CHECKLIST 

TESTING DATA COLLECTION INSTRUMENTS AND DATA COLLECTION SYSTEMS 



A. PURPOSE AND AUTHORITY 

1. Purpose of the questionnaire is clearly stated including a description of how 
the data are to be used. 

2. Clear, specific statements about who is requested to complete the 
questionnaire are included. 

3. Clear statements on the authority to collect the information and on the 
voluntary nature of participation are on the first page of the form. 

4. A Privacy Act statement and a statement regarding data confidentiality appear 
on the first page of the questionnaire. 

5. Reference is made to any groups officially supporting the survey. 

6. OMB clearance number is placed in the upper right-hand corner of the first 
page of the questionnaire. 

7. Expiration date, form number and edition dates are located on each page of 
the questionnaire. 

B. DATA REQUIREMENTS 

1. Each data element is needed to support the purpose of the survey. 

2. Time requirements for completing the questionnaire is not excessive (in most 
cases, less than 30-45 minutes). 

C . QUESTION CONTENT AND WORDING 

1. Question structure (i.e. open-ended, multiple choice, etc.) is the most 
suitable for information requested. 

2. Questions do not lead tt> biased answers. 

3. Wording is geared to the particular respondents. 

4. Double or compound questions are not used. 

5. Only commonly accepted abbreviations are used. 

6. Reference periods or dates are clearly specified. 

D . FORMATTING 

1. Questions on the same general topic are grouped together. 

2. Skip patterns, clearly identified, are used to ensure that only appropriate 
questions are asked -of each respondent. 

3. Complex skip patterns and ambiguous questions are avoided. 

4. Sufficient space is provided for responses. 

5. Use of check boxes and preprinted data is maximized. 

6. All items are numbered. 

E. INSTRUCTIONS 

1. All necessary definitions are provided. 

2. Instructions are provided for each item on the form where necessary to avoid 
ambiguities or uncertainties. 

3. Instructions for specific questions are numbered to correspond to the 
questions they explain and are located next to the question where feasible. 

4. Instructions for returning questionnaire are provided. 

5. A phone number is provided for respondent inquiries. 

- 10 - 
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CENTER FOR EDUCATION STATISTICS 
OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT 
U.S. DEPARTMENT OF EDUCATION 



SUBJECT: STANDARD FOR EDUCATIONAL TESTS 

EFFECTIVE DATE: 03/16/87 CES STANDARD 87-03-02 

PURPOSE: To ensure that educational tests produced by the CES meet professional 
standards of integrity. CES adopts as its standard the "primary standards" 
specified in Standards for Educational and Psychological Testing (American 
Psychological Association, 1985). The standards apply to performance tests, 
questionnaires (such as rating scales), and behavorial observations. 

o Evidence of validity should be presented for each recommended use of 
the test. 

o Evidence of reliability and standard errors of measurement should be 
presented for each recommended use of the test. 

o The procedure used to develop tests should be clearly specified. 

o Test administration should be standardized. 

o Standardized scoring and reporting procedures should be followed. 

o Special testing conditions should be arranged for special populations 
of examinees. 

o The rights of test takers should be protected. 



RELATED STANDARDS: 

o Standard for Planning Statistical Surveys, 86-01-01 

o Standard for Testing Data Collection Instruments, 87-03-01 

o Standard for Maintaining Confidentiality, 87-04-03 

RELATED CHECKLISTS, FORMS AND DOCUMENTATION: 

° Standar ds for Educational and Psychological Testing. American 
Psychological Association, 1985 

-11- 
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CHECKLIST 



EDUCATIONAL TESTS 



A. Evidence of Validity 

1. When a test measures a construct, evidence should be presented 
that the test scores are highly related to that construct (include 
inter- item correlations, bases for weighting subscorcs, 
theoretical basis for the construct, and a description of the 
construct-related validity study). 

2. If a test is purported to be related to an outcome criterion, then 
evidence of the relationship between the test score and the 
outcome measures must be provided. Information should be 
available related to the rationale for choosing criteria, 
justification for selecting cut-scores, and a description of the 
criterion-related validity study. 

3. When test content is especially important (as in achie\ ^t 
tests), content-related evidence of validity should be p*. ided. 
Clear definition for the content universe, domain specifications, 
item specifications, and a description of the content-related 
validity study should be provided. 

B. Evidence of Reliability and Standard Errors of Measurement 

1. Methods of calculating reliability coefficients and standard 
errors of measurement should be reported (include equations, and a 
description of the reliability study) . 

2. The psychometric procedures underlying the calculation of 
reliability indices should be specified (such as classical 
measurement theory, item response theory, etc.) 

3. Limitations in the reliability indices should be specified. For 
example, how does the index vary with changes over populations, 
the effects of adjustments for attenuation (e.g., due to 
restriction of the measurement range) and the effects of 
speededness (due to testing time limits) . 

C. Test Development Procedures 

1. Domain definitions and specifications should be provided. 
Similarly item and -test specifications should be clearly reported. 

2. Item selection procedures should be indicated (including 
statistical item performance data, item bias information, and 
psychometric criteria for selection.) 

D. Test Administration 

1. Administration procedures should be standardized to reduce 
unwanted variation in test scores. The same administration 
manuals, training and testing conditions should be used. 

-12- 
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CHECKLIST 
FOR EDUCATIONAL TESTS 



2. Steps should be taken to insure test security, proper distribution 
and collection of materials, and a minimum of respondent burden. 

E. Scoring and Reporting 

1. Scoring procedures should be standardized, documented and checked 
for accuracy, 

2. The analysis of item responses or test scores should be clearly 
described (such as scaling, norming, equating, etc.) 

3. Score reports should be promptly reported in a standard format 
that is easy to read. 

4. The generalizability and limitations of reported scores should ba 
presented. 

F. Special Testing Conditions 

1. Testing modifications should be provided for examinees with 
handicapping conditions or language differences. 

2. When feasible, the validity, reliability and other indices of test 
integrity should be investigated for special populations. 

G. Rights of Test-Takers 

1. Proper consent should be obtained prior to test administration. 

2. Rights of confidentiality should be guaranteed for all uses of the 
data. 
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CENTER FOR EDUCATION STATISTICS 
OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT 
U.S. DEPARTMENT OF EDUCATION 



SUBJECT: STANDARD FOR MINIMIZING SURVEY NONRESPONSE 

EFFECTIVE DATE: 03/16/87 CES STANDARD 87-03-03 

PURPOSE: To ensure that studies routinely perform those actions nec- 
essary to study the causes of and to minimize both survey and item 
nonresponse. 

o Nonresponse (or total nonresponse) refers to the failure of the 
survey respondent, or for administrative record studies, of the 
person or organization providing the data, to provide any 
information in response to the data request. Item nonresponse 
refers to certain items in the data request being missing when 
initially a response is expected. 

o For total nonresponse, in studies conducted by the Center or its 
contractors, very serious efforts should be made to collect 
information not obtained initially in the survey. Callbacks with 
the respondents or data providers will be scheduled during and 
immediately after the data collection period. Records on the number 
of callback attempts and their outcome should be kept, and a final 
disposition of the case (the reason it continues as a nonresponse 
case) should be kept in lieu of a completed interview or shuttle 
form. Attempts should be made to determine characteristics of the 
nonrespondents for xtse in weighting and imputation. 

o For item nonresponse, key statistics to be collected will be 

identified in the analysis plan; some provision should be made in 
the field operations to monitor the responses on these items to 
ensure that no one interviewer (in a survey) has an exceptionally 
high item nonresponse rate for key items. The project officer 
should also consider a field or office edit that would require 
telephone followup for cases with key items missing. 

o The survey design should consider explicitly the motivation of 
respondents, and the kinds of messages and publicity that might 
improve total response. 

o In addition, well-designed survey or shuttle forms and well-trained 
interviewers or clearly written survey (shuttle) instructions will 
be the most efficient devices for ensuring a high response. 
Procedures for minimizing nonresponse must be well documented. These 
procedures should be reviewed as part of and in conjuction with any 
pretest of the survey questionnaire or the use of shuttle forms for 
administrative record data collection. 

RELATED STANDARDS: 

o CES Standard for Planning Statistical Surveys, 86-01-01 

o CES Standard for Testing Data Collection Instruments, 87-03-01 

o CES Standard for Treatment of Item Nonresponse, 87-05-01 

-14- 
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CENTER FOR EDUCATION STATISTICS 
OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT 
U.S. DEPARTMENT OF EDUCATION 



SUBJECT: BENCHMARKS FOR D^a COLLECTION, PROCESSING, AND ANALYSIS 
EFFECTIVE DATE: 03/16/87 CES STANDARD 87-03-04 

PURPOSE: To establish minimum levels for performance in surveys and 
studies conducted by the Center. The levels of data completeness and 
minimum levels of data required for processing procedures and analysis are 
established to ensure that researchers and users will have confidence in 
the quality of the data. Benchmarks are reference points for judging 
quality. 

There are occasions when the benchmarks listed below are not appropriate 
to the study being conducted (e.g. certain types of experiments, pretests, 
or policy studies where a quick response is the highest priority) , In 
these cases, the researcher will kntfw in advance that the benchmarks are 
not appropriate to the study; the project officer should request an 
exception to the specific targets that he or she believes will not be 
met. This request must come in writing to the Chief Statistician and be 
part of the CES clearance process for the survey. The request must 
clearly state which targets or benchmarks will not be met, what the 
project officer's expectations are for the targets, and the reason why an 
exception should be granted. Reports for such studies must include in the 
technical appendices a statement about the deviation from the benchmarks 
including both the rationale for the deviation and the magnitude of the 
deviation. 

o The overa]" 1 survey target response rate specified in RFP's 
should be at least 90 percent for longitudinal surveys, 85 
percent for cross -sectional surveys. In the case where the 
sample is selected hierarchically (e.g. schools, and then 
teachers within those schools), these rates apply to each 
hierarchy (e.g. 85 percent of schools responding, and then 85 
percent of teachers within the responding schools, for an 
overall rate of 85% x 85% « 72.25%). Response rates for 
sample surveys are calculated on weighted data; response rates 
for census or administrative record data are based on 
unweighted data. 

o Within any stratum of a sample, the overall survey target 
response rate should be no less than 85 percent for 
longitudinal surveys and 80 percent for cross-sectional 
surveys. In the case where the sample is selected 
hierarchically, these ratas apply to each hierarchy. Response 
rates for sample surveys are calculated on weighted data; 
response rates for census or administrative record data are 
based on unweighted data. 
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BENCHMARKS FOR DATA COLLECTION, PROCESSING, 
AND ANALYSIS 



o The target item response rate for each critical variable 

should be at least 85 percent (critical variables are defined 
in the analysis plan for the study). Response rates for 
sample surveys are calculated on weighted data; response rates 
for census or administrative record data are based on 
unweighted data. 

o Deviations from the benchmark figures given above should be 
anticipated in the planning phase of a survey. If the project 
officer expects the deviations to be severe, they should be 
documented in the analysis plan with a proposal as to how to 
minimize problems before they happen in the survey and a proposal 
regarding how the analysis of the data will adjust for deviations 
that cannot be overcome by the survey design. 

o If response rates and item response rates are lower than 

anticipated and fall below the benchmark levels, an analysis of 
the reasons for the low rates and the anticipated impact on the 
quality of the data must be conducted before any analysis of the 
survey data is done. 

o Variables with more than 30 percent missing data should not be 
used in analysis, except in tabulations where the missing data 
are tabulated as a separate category and clearly identified. 

o Weighting of sample data for nonresponse, when done by strata 
or within cells so that the weighting factors are calculated 
as ratios, should be based on a minimum of 30 respondents 
(unweighted) per cell. Cells that have fewer than 30 
respondents should be collapsed with the "closest" cell with the 
fewest number of respondents. "Closest" is defined as logically 
closest, i.e. contiguous age categories or contiguous 
geographic areas that make substantive sense. 

o Weighting of sample data for nonresponse should also be done 
with caution. The ratio of the largest stratum nonresponse 
weighting factor to the smallest statum nonresponse weighting 
factor should be no more than 5. In cases where the ratio is 
larger than this, the smallest stratum in terms of unweighted 
respondents should be collapsed with the "closest" cell with 
the fewest number of respondents. 

o Estimates of means, proportions, and totals should be computed 
from at least 30 respondents for each subgroup for which the 
estimate is made; estimates of ratios, rates, regression 
coefficients (in cases where pairwise deletion methods are 
used in computation of the correlation matrix), and similar 
multivariate statistics should be based on at least 30 
respondents in both the numerator and denominator of each 
statistic when the data used come from a survey or the 
numerator and denominator come from different data sources, at 
least one of which is a survey. These minimums do not apply 
when the data come from a census or administrative record 
study not involving sampling. 
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BENCHMARKS FOR DATA COLLECTION, PROCESSING, 
AND ANALYSIS 

Confidence levels for any results of statistical tests 
reported in a document should be at least 90 percent before 
the null hypothesis is rejected. 

Confidence intervals around key statistics (as defined in the 
analysis plan) reported in a document or a table should be 95 
percent confidence intervals and should be clearly identified 
as such . 

There should not be more than 20 "simple" comparisons made 
within a bulletin or a report. "Simple" is defined as a 
t-test, chi-square test, or any other test that examines a 
simple hypothesis like the difference of means or proportions. 
Consideration must be given to use of multivariate techniques 
in analyses involving multiple variables, factors, or levels, 
and/or an analysis of overall error rates should be conducted 
where multiple comparisons and univariate variables are used. 

Overall response rates (Ro) are to be calculated as the 
ratio of the number of completed interviews (see CES Standard 
87-05-01) divided by the number of sample respondents drawn minus 
respondents considered out-of-scope (in a household interview 
this would be number of units sampled minus vacant units, 
condemned units, or units that have been converted from 
residential to business use): 

weighted # of completed interviews 

RO m 

weighted # of units sampled - weighted # of out-of -scopes 

Item response rates (Ri) are to be calculated as the ratio of 
the number of respondents for which an in-scope response was 
obtained (i.e. the response conformed to acceptable categories 
or ranges) divided by the number of completed interviews for 
which the question (or questions if a composite variable) was 
intended to be asked: 

weighted # of respondents with in-scope response 
Ri - -- „_ 

weighted # of completed interviews for which question was 
intended to be asked 

When the items being studied are continuous or additive (e.g. 
number of teachers is discrete but additive, whereas 
affiliation categories are not) , coverage rates should also be 
calculated. Coverage rates describe the relative loss of 
information because of size of the unit which gave an 
incomplete response, not the number not responding. 

weighted total of in-scope responses 
Ci - 

weighted total of completed interviews for which question 
was intended to be asked (includes imputed values) 



BENCHMARKS FOR DATA COLLECTION, PROCESSING, 
AND ANALYSIS 



RELATED STANDARDS: 

o CES Standard for Planning. Statistical Surveys, 86-01-01 
o CES Standard for Testing Data Collection Instruments, 87-03-01 
o CES Standard for Minimizing Survey Nonresponse, 87-03-03 
o CES Standard for Treatment of Item Nonresponse, 87-05-01 
o CES Standard for Technical Documentation in Data Releaser, 
86-04-02 
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CENTER FOR EDUCATION STATISTICS 
OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT OF PRACTICE 
U.S. DEPARTMENT OF EDUCATION 



SUBJECT: STANDARD FOR CODES, ABBREVIATIONS A><D ACRONYMS 

EFFECTIVE DATE: 03/16/87 CES STANDARD 87-03-05 

PURPOSE: To provide uniform codes, abbreviations and acronyms for use in CES 
data collection and processing In order to facilitate the exchange of 
information and to ensure uniformity in CES publications. 

o The following Federal Information Processing Standards (FIPS PUBS) are to 
be used in all CES publications: 

- 5-1 States and Outlying Areas of the U.S. 

- 6-3 County and County Equivalent of the States of the U.S. and D.C. 

- 8-5 Standard Metropolis u Statistical Areas 

- 9 Congressional Districts of the U.S. 

- 66 Standard Industrial Classification Codes (SIC) 

- 92 Standard Occupational Codes (SOC) 

If it is not appropriate to use these codes, a justification In writing to 
the Forms Clearance Officer is needed as a part of the CES clearance 
process for the survey. 

o The variables of "race" and "ethnic origin" are generally treated as two 
separate population characteristics. 0MB has specified four minimum 
categories or race designation to be used in collecting and presenting 
data: 

- American Indian or Alaskan Native 

- Asian or Pacific Islander 

- Black 

- White 

o The designation "nonwhite" is not acceptable for use in the presentation of 
Federal Government data, and the term "color" is not acceptable for use by 
OERI. Therefore, the designations "black and other races" and "all other 
races" are used as collective descriptions of minority racas when the most 
summary distinction between the majority and minority race is appropriate. 
"White," "black" and "all other races" are used when the distinction among 
the majority race, the principal minority race, and other races is 
appropriate. Similarly, the designation of a particular minority race or 
races with "white" and "all other races" is used ir such a collective 
description is appropriate. 

o Statistical data collections that include a question on race also should 
include a question to determine, at a minimum, whether an individual is or 
is not of Hispanic origin. The minimum detail expressed on ethnic 
background presented in tables must include "total," "Hispanic origin" and 
"non-Hispanic origin." More detailed groups must be organized in such a 
way that the additional categories can be aggregated into either Hispanic 
or non-Hispanic. 
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STANDARD FOR CODES, ABBREVIATIONS AND ACRONYMS - (Continued) 



o The approved list of acronyms in the OERI Editorial Stylebook is to be used 
in all CES publications. 

o CES manuals and glossaries of definitions should be used in all CES surveys 
and publications. Pending a revision of this standard, which will include 
a dictionary of terms, glossaries must also be submitted for clearance. 

o Official national, State and international abbreviations are listed on 
pages 135-160 of the Style Manual of the U.S. Government Printing Office. 
These abbreviations will be used where appropriate in CES publications. 

RELATED STANDARDS 

o CES Standard for Planning Statistical Surveys, 86-01-01 
o CES Standard for RFP Development for Surveys, 87-02-01 
o CES Standard for Survey Documentation, 86-04-01 

o CES Standard for Technical Documentation in Data Release^, 86-04-02 
RELATED FORMS, CHECKLISTS AND PUBLICATIONS 
o OERI Editorial Stylebook 

o Federal Information Processing Standards Publications (FIPS PUBS) 
o Race and Ethnic Standards for Federal Statistics and Administrative 

Reporting, Office of Management and Budget 
o Style Manual . U.S. Government Printing Office, 1984 
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CENTER FOR EDUCATION STATISTICS 
OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT 
U. S. DEPARTMENT OF EDUCATION 



SUBJECT: STANDARD FOR ROUNDING 

EFFECTIVE DATE: 02/09/87 CES STANDARD 87-03-06 

URPOSE: To assist researchers in attempting to summarize and study the data 
from a survey. 

o Before rounding figures for publication, decision must be made about the 
appropriate number of significant digits. The problem of implying a 
spurious degree of accuracy must be balanced with the need to have 
consistency between and within tables. 

o When adding, multiplying or dividing figures that have been rounded to 
different significant digits, the product can only be stated in terms of 
the number with the fewest significant digits. (If 4.5 and 5.75 are 
rounded numbers, the product can be stated only as 26, with 4.5 having 
two significant digits and 5.75 having three.) 

o Tabulations to produce summary data and computations performed for 
purposes of estimating standard errors should be done on unrounded 
numbers . 

o Stuns of column (row) figures in a table should be derived using 
unrounded figures, with appropriate rounding of the total after its 
derivation. To handle the problem of column(row) figures not summing to 
a rounded total, an explanatory footnote should be used. 

o The final rounded value should be obtained from the original figure 
available, not from a series of roundings (e.g. 7.1748 can be 7.175 or 
7.17 or 7.2 or 7 but not 7.18). 

o Percentages should generally be rounded to no more than one decimal 
place. 

o Specific rules to be used for rounding: 

If the first digit to be dropped is less than 5, the last retained 
digit is not changed. 

6.1243 6.124 

If the first digit to be dropped is greater than or equal to 5, the 
last digit retained is increased by 1. 

6.688 6.69 

o Documents need footnotes explaining cases where actual numbers are used 
in all calculations, and rounded results are presented which seem 
inconsistent . 

RELATED STANDARDS: 

RELATED CHECKLISTS , FORtiS* AND DOCUMENTS : 
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OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT 
U.S. DEPARTMENT OF EDUCATION 



SUBJECT: STANDARD FOR DATA TAPE PREPARATION 

EFFECTIVE DATE: 12/08/86 CES STANDARD 86-03-07 

PURPOSE: To ensure that master data files prepared by CES staff and by- 
contractors are fully useable by researchers in and outside the Center. 

o Data Set Names should Identify the survey system, component, year and 
version number. 

o Tapes stored at the Department's central computer facility must be 

9-track EBCDIC, 6250 BPI with IBM standard labels. Tapes In different 
formats will be produced to meet user requests. 

o File structure, record format and data element conventions must be 
consistent with CES checklist on "Data Tape Preparation. 11 

o OERI Form 27, Data Base Documentation, must be completed for all tapes. 
This includes: 

Abstract or summary 

Survey methodology including universe and sample, 
respondents, dates of coverage, frequency counts 
and response rates 

File description 

Record length, format and count 

Data element definitions 

o A machine readable file with a file description and record layout 
information must be copied on all tapes prepared for public 
distribution. 

o If a SAS system file is prepared, appropriate documentation must be 
developed. 



RELATED STANDARDS: 

o CES Standard for Survey Documentation, 86-04-01 

o CES Standard for Technical Documentation in Data Releases, 86-04-02 

RELATED CHECKLISTS AND FORMS: 

o Data Tape Preparation Checklist, 87-03-07 
o OERI Form 27: Data Base Documentation 
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CHECKLIST 
DATA TAPE PREPARATION 



A, Data set names give the survey system, component, year and version, 
B Standard Definitions 

1. FIPS Standards are used where applicable, 

2. CES standard definitions and codes are used where applicable, 

C, Data Element Conventions 

1, Numeric fields contain only numbers, zeros or missing values, 

2, Zeros are represented by n p n . Do not use a "-" or blank, 

3, Suppression symbols are removed from numeric fields and stored in 
associated "flag" fields. 

4, Negative numbers are represented in standard (IBM) signed numeric 
format. 

5, When practical, numeric data fields containing continuous variables 
should be identical in length, 

6 , Separate record locations are used for all data items . 

D. Record Conventions 

1. Rectangular structures are used, where possible, 

2. Single record type is used. 

3. Logical record lengths are constant. 

4. Each record contains a unique identifier such as ID. 

E. Record Layout 

1, Record layout, location and format are documented. 

2. Blocking factors and block size are documented. 

3, Record counts are included, 

4. Each record type is identified, 

F, Tape Format 

1. 9 track, EBCDIC, 6250 BPI standard label tapes are used. 

2. Standard alphanumeric characters are used. 

3. Binary or packed decimal fields are avoided unless required by the size 
of the file or other factors, 

6, Quality Control Procedures 

1, Record counts and totals are consistent between preliminary files and 
the final released tapes or explanations are included in the 
documentation • 

2, Internal inconsistencies are corrected or documented, 

3, Major inconsistencies between current and past data are resolved or 
documented. 

4, Imputed data are flagged. 

5, Flags are consistently positioned within a file. 

6, Codes for missing, refused and non-applicable data are consistent 
within a survey or survey system. 

7, All necessary steps have been taken to protect the confidentiality of 
respondents . 
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CENTER FOR EDUCATION STATISTICS 
OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT 
U.S. DEPARTMENT OF EDUCATION 



SUBJECT: STANDARD FOR SURVEY DOCUMENTATION 
EFFECTIVE DATE: 12/08/86 



CES STANDARD 86-04-01 



PURPOSE: To ensure that data released by CES ere documented in a consistent 
manner . 

Documentation must allow the non- statistical user to understand the limitations 
and quality of the data. Researchers must have more specific and detailed 
documentation of the data to permit independent verification and analysis. 

Written detailed documentation must be available on all major surveys conducted 
or supported by CES. Topics to be included are provided on the attached 
checklist, and summarized below: 

o Abstract: purpose, target population, time and geographic coverage, and 
contact person or office. 

o Status of data: current status (preliminary, revised or final) and 
relation to previous surveys in series; 

o Methodology and data preparation: definitions, sample selection, data 
collection process, data reduction and estimation, extent of editing and 
its effects; 

o Sampling and non- sampling errors: methods for estimating and adjusting 
for errors. 

o Guidelines for using the data: major caveats, use of weights, use of 
statistical packages, related surveys and research. 



RELATED STANDARDS 

o CES Standard for Data Tape Preparation, 87-03-07 

o CES Standard for Technical Documentation in Data Releases , 86-04-02 
RELATED CHECKLISTS, FORMS AND DOCUMENTS 

o Survey Documentation Checklist, 86-04-01 
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CHECKLIST 
SURVEY DOCUMENTATION 



A. Abstract 

1. The history, purpose and legislative mandates for the survey are 
clearly explained. 

2. The target population Is described. 

3. The time and geographic coverages of the survey are described. 

4. Contact persons (s) or relevant office are named with current phone 
numbers . 

B. Status of Data 

1. The data are clearly Identified as preliminary, test, revised or final. 

2. A schedule of revisions Is included, where applicable. 

3. Differences between this survey and previous surveys in the same series 
are explained. 

C. Methodology and Data Preparation 

1. Definitions 

a. Technical definitions are Included. 

b. Concepts critical to understanding data are defined. 

2. Sample Design 

a. The target population Is described. 

b. The survey frame is described with regard to the source of the 
frame, reference date and number of units. 

c. The units selected for the sample at each stage are Identified. 

d. The numbers of sampling units at each stage are provided. 

e. The procedures for allocating the sample at each stage are 
discussed. 

f . The sample selection process at each stage is described. 

g. The response rates and their derivations are provided. 

h. Measures of size defined for sampling with "probability 
proportionate to size" are provided. 

I. Improvements in precision that result from the choice of a 
particular sample design are discussed. 

3. Data Collection Process 

a. There Is a general description of how the instrument (s) were 
administered. 

b. Copies of the interview forms/questionnaires are included. 
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CHECKLIST FOR SURVEY DOCUMENTATION - (Continued) 



c. Quality control procedures used in the data collection process and 
the results of the implementation of these procedures are 
described* 

d. Results of pretests and independent evaluations are discussed* 

4. Data Reduction and Estimation 

a. The derivations of the base weights and any special weights are 
specified* 

b. Estimation procedures are described. 

c. Procedures used to adjust for non-response are described. 

5. Editing and Imputation 

a. Methods for imputing data are described. 

b. Procedures for editing data are described. 

c. Problems or limitations that resulted from edit procedures are 
described. 

D. Sampling and Non-Sampling Error 

1. Sampling Errors 

a. Methods for estimating sampling errors are discussed. 

b. Generalized or specific sampling error tables are provided. 

c. Guidance is provided on how to apply tables of sampling errors to 
the errors In reported statistics. 

2. Non-sampling Errors 

a. Response levels as well as methods for adjusting fo^ non-response 
are discussed. 

b. Response bias and measurement errors are discussed. 

c. Conceptual and other limitations of the data are pointed out. 

E. Guidelines for Using the Data 

1. Caveats are included about how data can and cannot be used. 

2. Instructions are provided on which sample weights to use. 

3. Control counts, tables or distributions for critical variables are 
provided. 

4. Confidence levels are appropriate and explained. 

5. Design effects are provided, where appropriate. 

6. Any special considerations for using statistical packages such as SAS 
or SPSS are included. 

7. Related surveys and studies are listed in a bibliography. 
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CENTER FOR EDUCATION STATISTICS 
OFFICE OF EDUCATION RESEARCH AND IMPROVEMENT 
U.S. DEPARTMENT OF EDUCATION 



SUBJECT: STANDARD FOR TECHNICAL DOCUMENTATION IN DATA RELEASES 

EFFECTIVE DATE: 12/08/86 CES STANDARD 86-04-02 

PURPOSE: To ensure that reports, bulletins, press announcements and data 
tapes released by the CES accurately report results of surveys and other 
studies conducted by CES In such a way that the reader can evaluate the 
conclusions drawn. 

The level of documentation, length and technical nature of the discussion will 
vary with type of publication and the audiences for whom the publication is 
intended. 

o Major reports on surveys prepared by contractors or by CES staff for 

publication must contain or refer to a technical appendix that discusses 
all topics required in CES Standard for Survey Documentation. This 
includes: 

summary or abstract of when the data were collected, target 
population and purposes of the survey; 

- a statement on the status of the data (preliminary, revised or 
final) ; 

- a discussion of the methodology used; 

- a discussion of sampling errors including how the sampling errors 
were calculated and tables of sampling errors If not in the text; 

- a discussion of non-sampling errors Including sources of error, what 
was done to control these sources and any external corroborating 
evidence used to validate the data; and 

- guidelines for using the data. 

o All key statistics In technical reports should be followed by confidence 
intervals. Where comparisons are made between two or more statistics, a 
statement should be made as to the significance of the test .performed 
(e.g. "significant at the 95% confidence level" or "alpha less than 
.05"). The test used must be cited. If the same test Is used throughout 
the report, this should be stated at the beginning of the report and 
then exceptions noted at relevant points . 

o Short reports and bulletins must have a statement about the status of 
the data, sample selection, data collection process, the sampling error 
for key statistics presented, or alternatively, present confidence 
Intervals after the key statistics. The report must eference more 
detailed documentation and give the name and phone number of a contact 
person or office. 
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STANDARD FOR TECHNICAL DOCUMENTATION IK DA 7 fA RELEASES - (Continued) 



o Press releases, one or two-page bulletins, and responses to ad hoc 
requests must give the status of the data and include a statement 
indicating the sampling error for key statistics, 

o Public release data tapes must conform to CES Standard for Data Tape 
Preparation including documentation of file structure, record size, 
count and blocking factors, data element definitions and locations. The 
data set name (DSN) must refer to the survey name, the series if 
appropriate, and the version or release date, 

o E.D. TABS and other tabulations prepared in either print, tape or disk 
format, must discuss status of data, sample design, data collection 
procedures, salient definitions, response rates, type of data collected, 
sampling and non-sampling errors. 

o Proper documentation must be provided for any data reported from a 
source other than the survey including source name, publication 
information, table number and page. 



RELATED STANDARDS: 

o CES Standard for Survey Documentation, 86-04-01 

o CES Standard for Codes, Abbreviations and Acronyms, 87-03-05 

o CES Standard for Data Tape Preparation, 87-03-07 

o CES Standard for Estimating Sampling Error, 87-05-02 

o OERI Publication Policy 

o CES Publication Policy 

RELATED CHECKLISTS, FORMS AND DOCUMENTS: 

o Maria E. Gonzalez, Jack L. Ogus, Gary Shapiro and Benjamin J. "Tepping, 
"Standards for Discussion and Presentation of Errors in Survey and 
Census Data," Journal of the American Statistical Association, 
September, 1975, Volume 70, Number 351, Part II. 
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CENTER FOR EDUCATION STATISTICS 
OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT 
U. S. DEPARTMENT OF EDUCATION 



SUBJECT: STANDARD FOR MAINTAINING CONFIDENTIALITY 

EFFECTIVE DATE: 02/09/87 CES STANDARD 87-04-03 

PURPOSE: To ensure that the rights of individuals responding to CES surveys 
are protected and that confidentiality of responses is maintained. 

o As required by 0MB, respondents must be told whether or not 

participation in the survey is voluntary and about the legislative 
or executive mandate for the survey. 

o Respondents must be told in a cover letter or in instructions 
whether responses will be kept confidential and what provisions 
will be made to ensure this confidentiality. 

o In developing RFPs for surveys, Project Officers must describe the 
extent to which confidentiality must be maintained. The more 
sensitive the data, the more it is important that voluntary 
participants know that individual responses will be not 
identifiable. 

o Where absolute confidentiality has been promised or is required by 
law, Project Officers should not physically accept from a 
contractor a document or tape that gives the identification of 
individuals . 

o In reporting on surveys and preparing data tapes, care must be 
taken to ensure that individual respondents cannot be identified 
where confidentiality has been promised. If the nonzero value of 
a column, row or total is based on information from one or two 
respondents, the cell is considered sensitive and must be 
suppressed in such ways that the cell cannot be constructed from 
values in other cells. Suppressed cells must be flagged. 



RELATED STANDARDS: 

o CES Standard for Planning Statistical Surveys, 86-01-01 

o CES Standard for RFP Development for Surveys, 87-02-01 

o CES Standard for Monitoring Survey Contracts, 87-02-02 

o CES Standard for Standard for Survey Documentation, 86-04-01 

RELATED CHECKLISTS, FORMS AND DOCUMENTS: 
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CENTER FOR EDUCATION STATISTICS 
OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT 
U. S. DEPARTMENT OF EDUCATION 



SUBJECT: STANDARD FOR TABULAR PRESENTATION 

EFFECTIVE DATE: CES STANDARD 87-04-04 

PURPOSE: To ensure that tables used In CES publications display data In such a 
way that readers can evaluate the accuracy and significance of the results 
presented. 

o All tables must have a short title concisely stating the subject of the 
table and the relevant time period, 

o All variables used in a table, whether bivariate or multivariate, must 
be labeled with the name of the variable and with categories clearly 
labeled. 

o When data presented in a table come from multiple sources or come from a 
source which is not the direct subject of the report, the sources must 
clearly be cited at the bottom of the table. Multiple sources must be 
footnoted so the reader knows exactly which survey or report was the 
source of the data. 

o All row and column totals must be presented in tables. 

o For tables of percentages or rates, population sizes must be presented 
at the bottom of the table. If different rows or columns of the table 
are based on different population sizes, each population size must be 
presented at the bottom of the table. 

o Standard errors (se's) or confidence intervals (ci's) on statistics in 
tables can be handled in two ways. The first and preferred method is to 
include se's or ci's in the table being presented, either in a separate 
subsection of the table or in columns accompanying the statistics being 
presented. The second method is to have a separate table of standard 
errors on the statistics being presented which follows the tables in the 
report . 

RELATED STANDARDS: 

o CES Standard for Dissemination of Survey Data and Results ,87-04-05 
o CES Standard for Estimating Sampling Error, 87-05-02 

RELATED CHECKLISTS, FORMS AND DOCUMENTS: 

o Suggestions for Use of Graphics, p. 31 
o NCES Guidelines for Tabular Presentation 

o James Filliben and David DesJardins, The Graphic Analysis and Display of 
Statistical Data , Washington Statistical Society, 1986. 

o Calvin F. Schmid and Stanton E. Schmid, Handbook of Graphic 
Presentation . 1979. 

o Edward R. Tufte, The Visual Display of Quantitative Information . 
Graphics Press, 1983. 
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CENTER FOR EDUCATION STATISTICS 
OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT 
U. S. DEPARTMENT OF EDUCATION 



SUBJECT: STANDARD FOR DISSEMINATION OF SURVEY DATA AND RESULTS 



PURPOSE: To ensure that the public have the data and results of CES surveys in 
a predictable and timely fashion. 



o Planning for dissemination of results of a survey must be done 
concurrently with planning for the survey Itself, i.e., in the 
beginning . 

o Major findings from recurrent and annual data collection efforts 
will be published on a fixed schedule so that the public will know 
when to expect these data series from year to year. 

o Publications, data tapes and analyses must be ready for 
distribution on planned dates. 

o The shortest practical Interval should exist between the date or 
period to which the data refer and the date when a clean date tape 
is completed. A clean data tape should be ready within 6 months 
of completion of data collection. 

o Project Officers are encouraged to prepare articles for journals 
and to make presentations at professional meetings. 

o Publications and data tapes released by CES will be 100% accurate 
in terms of consistency with master data files and internal 
consistency (e.g. columns and rows total) . Appropriate technical 
documentation, consistent with CES Standard for Technical 
Documentation in Data Releases will be followed. 



o CES Standard for Technical Documentation in Data Releases, 
86-04-02 

o Standard for Timely Processing and Release of Data and Data Tapes, 

97-04-06 
o OERI Publication Policy 
o CES Publication Policy 



EFFECTIVE DATE: 02/09/87 



CES STANDARD 87-04-05 



RELATED STANDARDS: 



RELATED CHECKLISTS, FORMS AND DOCUMENTS: 
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CENTER FOR EDUCATION STATISTICS 
OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT 
U. S. DEPARTMENT OF EDUCATION 



SUBJECT: STANDARD FOR TIMELY PROCESSING AND RELEASE OF DATA AND DATA TAPES 
EFFECTIVE DATE: 02/09/87 CES STANDARD 87-04-06 

PURPOSE: To ensure that data, tapes and reports from CES are released in a 
timely, scheduled manner. 

o Data from recurrent surveys and annual publications will be released 
according to a published schedule. If changes in the published schedule 
become necessary, they must be cleared with the CES Director and 
announced at least 30 days in advance of the originally scheduled date 
of publication. 

o To meet publication deadlines, other deadlines and milestones in the 
survey process must also be met including timely completion of planning, 
issuance of the RFP, review or proposals and awarding of contracts, and 
review of deliverables. 

o To help minimize the number of non-responding units by publication time, 
all units (e.g. State departments of education) should be notified of 
the scheduled release date and the fact that the data will be reported 
on that day with or without their information. 

o The shortest practical interval should exist between the date or period 
to which the data refer and the date when a clean data tape is 
completed. A clean data tape should be ready within 6 months of 
completion of data collection. 

o E . D . TABS or another printed report should be scheduled for publication 
two months after availability of a data tape. 

o Release of a data file to Information Services should be scheduled for 
no more than twenty working days after expected production of a clean 
data tape with documentation. After official approval by the Office of 
the Director, a data tape may be released to the public with a copy of 
technical documentation if final published documentation is not ready. 

o Data and data tapes must not be released to anyone or any group before 
official approval and release by the Office of the Director. 



RELATED STANDARDS: 

o CES Standard for Technical Documentation in Data Releases, 86-04-02 

o CES Standard for Dissemination of Survey Data and Results, 87-04-05 
o CES Standard for Treatment of Item Nonresponra, 87-05-01 
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CENTER FOR EDUCATION STATISTICS 
OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT 
U.S. DEPARTMENT OF EDUCATION 



SUBJECT: STANDARD FOR RELEASE OF STATISTICAL DATA 
EFFECTIVE DATE: 02/09/87 



CES STANDARD 87-04-07 



PURPOSE: To ensure that CES data are released without undue delay, on a 
publicly released schedule and with the minimum nv^ tx er of revisions. 

o Data may not be released in any form prior to official clearance and 
sign-off by the Office of the Director. 

o The release date for publications, data tapes and other data releases 
must be scheduled and available to the public . 

o The goal should be to have no more than two revisions of the data set 
for a particular survey series. 

o Only data from official CES released data tapes may be used in CES 

publications. The survey title, series and version must be referenced. 

o A released file must be kept with appropriate documentation at the 
designated Education Department Computer Center. Data set names must 
give survey, series and version number. Divisions are responsible for 
implementing procedures to ensure that tapes are maintained at the 
central Education Department computer facility. 

o Working data files used to produce published statistics must be archived 
for at least eighteen months. These files, together with the appropriate 
documentation, should be sufficient to allow an experienced computer 
programmer/analyst to duplicate the numbers released. 



RELATED STANDARDS: 

o CES Standard for Data Tape Preparation, 87-03-07 

o CES Standard for Technical Documentation in Data Releases ,86-04-02 
o CES Standard for Timely Processing and Release of Data and Data Tapes, 
87-04-06 

RELATED CHECKLISTS, FORMS AND DOCUMENTS: 

o "Statistical Policy Directive on Compilation, Release and Evaluation of 
Principal Federal Economic Indicators." Office of Federal Statistical 
Policy and Standards, Office of Management and Budget, September, 1985. 
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CENTER FOR EDUCATION STATISTICS 
OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT 
U. S. DEPARTMENT OF EDUCATION 



SUBJECT: STANDARD FOR TREATMENT OF ITEM NONRESPONSE 

EFFECTIVE DATE: 03/16/87 CES STANDARD 87-05-01 

PURPOSE: To ensure consistency of data over time and to ensure Internal 
consistency of data before analysis. 

o Best efforts should be made to minimize the occurrence of Item 

nonresponse. Normally, adequate pre- and pilot testing will Identify 
likely item nonresponse problems and protect against them. Geniunely 
unanticipated problems are to be dealt with post facto. The respondents 
should be contacted to complete the questionnaire before resorting to 
use of weighting or Imputation. 

o If more than thirty percent of the key items In a record are missing, 
consideration should be given to deleting the record and making a 
weighting adjustment for survey nonresponse, rather than imputing items. 

o If less than thirty percent of the key items In a record are missing, 
the missing items should be imputed. The type of imputation will depend 
on the type of survey and the type of data. 

o Any imputation on a data tape should be clearly indicated by a separate 
field with a flag set to Indicate the data values which have been 
imputed. 

o For continuing or longitudinal surveys where categorical data 
have been gathered In the past on the same Items to be imputed, 
missing values should be Imputed directly from the previous 
survey. 

o For continuing or longitudinal surveys where continuous data have been 
gathered in the past on the same items to be Imputed, a previous 
observation for the same unit Is available, and no strong trends are 
perceived in the data set, missing values should be Imputed directly 
from the previous survey. A strong trend Is defined as one where a 
linear or curvilinear regression has at least one coefficient 
significantly different from zero at the 95 percent confidence level. 

o For continuing or longitudinal surveys where two or more rounds of 
continuous data have been gathered in the past on the same Items to be 
imputed, at least one previous observation for the same item for this 
same unit is available, and strong trends are perceived in the overall 
data set for this item, missing values should be imputed by estimating 
the trend using linear regression on the data collected in the previous 
time period and data in the current time period. The regression would 
be run only on cases where observations were present in both time 
periods, and a separate regression would be run for each variable to be 
Imputed. 




For continuing or longitudinal surveys where continuous data are not 
available for more than 2 time periods, but a previous observation for 
the same unit is available, a ratio factor can be computed from those 
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STANDARD FOR TREATMENT OF ITEM NONRESPONSE (Continued) 



cases where observations are available in both time periods. This can 
be used to inflate or deflate prior observations to provide an Imputed 
value for missing data. Alternatively, if an inflation factor is known 
from outside sources (like CPI) , It can also be used. Ratios computed 
from the data are preferable, 

o For longitudinal surveys where previous observations are not available, 
or for one-time surveys, missing Items should be Imputed using the best 
practical estimator available. Normally, this means using hot deck, 
typically nearest neighbor, or other techniques that, for each 
imputation case, assign a donor case that is as similar as possible in 
terms of the characteristics being used to Impute, 

o Zero-filling a record because of missing data, or zero filling certain 
items which are missing, should only be used as a last resort, and 
reasons for these actions should be documented with the data tape. 



RELATED STANDARDS: 

o CES Standard for Planning Statistical Surveys, 86-01-01 

CES Standard, for Testing Data Collection Instruments, 87-03-01 
o CES Standard for Minimizing Survey Nonresponse, 87-03-03 

RELATED CHECKLISTS AND FORMS: 

o Kalton, Graham, and Kasprzyk, "Imputing for Missing Survey Responses, 
1982 Proceedings, American Statistical Association, p. 22-31. 
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CENTER FOR EDUCATION STATISTICS 
OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT 
U. S. DEPARTMENT OF EDUCATION 



SUBJECT: STANDARD FOR ESTIMATING SAMPLING ERROR 

EFFECTIVE DATE: 03/16/87 CES STANDARD 87-05-02 

PURPOSE: To establish procedures for developing estimation methods that can be 
used to generate standard errors for key statistics to be studied or reported 
in CES publications. 

o During the sample design phase for any survey to be conducted by or for 
the Center, consideration should be given to the methods to be used for 
variance estimation. If variances are to be calculated directly on the 
statistics to be studied, the variance estimators should be developed as 
part of the sample design. If variances are not calculated directly, 
but instead are calculated using a replication technique, the formation 
of replicates should be done as part of the sample design and the 
variance estimators that use the replicates should be developed as part 
of the sample design. 

o Where possible, estimation should make use of other data from the 

survey, from prior surveys, or from administrative records or censuses 
to minimize the variance of the survey estimates. Ratio or regression 
estimators should be used whenever possible if they do not substantially 
complicate the analysis while providing some reduction in the variance 
of the statistics being estimated. 

o If replication techniques are used, some research should be done as part 
of the sample design to determine the number of random groups to be 
used, and the stability of the estimators for the statistics being 
estimated. As a rule of thumb, jackknife estimators and balanced 
repeated replication are to be preferred over random groups estimators 
because of their greater stability in most cases, and the fact that 
jackknife estimators can eliminate the first order bias in ratio or 
regression estimates. 

o As part of the reporting of survey results, if there are a large number 
of statistics being estimated and presented, it may be more practical to 
model the variances and present the summary form of the model. This can 
be done by presenting the relationships between the relative variances 
and the values being estimated as a hyperbola called a GATT curve. A 
regression is run with the relative variance as the dependent variable 
and the reciprocal of the estimate for which the variance is calculated 
as the independent variable. A broad selection of variables to be 
reported on in the survey are used in the regression to establish the 
relationship, and different GATT curves can be calculated for different 
subgroups of the population to reflect both different sauple sizes and 
different design effects. The closeness of the fit of the GATT curve 
should be tested before the GATT curve is reported in a publication - a 
poorly fitting curve gives a bad approximation to the actual variances 
and should not be reported. 
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STANDARD FOR ESTIMATING SAMPLING ERROR (Continued) 



RELATED STANDARDS: 

o CES Standard for Planning Statistical Surveys, 86-01-01 

o CES Standard for Technical Documentation in Data Releases ,86-04-02 

o CES Standard for Tabular Presentation, 87-04-04 

RELATED CHECKLISTS, FORMS, AND DOCUMENTS: 
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CENTER FOR EDUCATION STATISTICS, OERI 



General Education Provisions Act (GEPA), Section 406 
General Education Provisions Act (GEPA), Section 405 



- CES mandate 

- NAEP 



Carl D. Perkins Vocational Education Act, Part C, Sections 421, 

422, 423 - Vocational 
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GENERAL EDUCATION PROVISIONS ACT (GEPA) 
as amended 
Section 406 



Center for Education Statistics Legislation 



General Education Provisions Act (GEPA) , as amended * Section 406 



Director" 



if feasible, on 
a State-by-State 
basis 



tfrtnmftt CENTER FOR EDUCATION STATISTICS 

Sec, 406, (a) There is established, within the Office of ^tli c Asaialr Educational Research 
ant Secretary; a Na t i o nal Center for Education Statistics (hereafter and inmrovement- 
in this sect ion referred to as the "Center"). The Center shall be Wovenent 
headed by tan Admiflia fc i - afc o i who shall be appointed by the Assist- 
ant Secretary in accordance with the provisions of title 5, United 

States Code, relating to appointments in the comp etitive service. 

(b) The purpose of the Center shall be to coUecdanddlssei^n^ " and a ^alvze 
statistics and other data related to education in tEe United States ' analyze, 
and in other nations. The Center shall— 



"Department 
Education 



of 



Uncollect, collate, and, from time to time, report full and 
complete statistics on the conditions of education in the United 
States; 

(2) conduct and publish reports on specialized analyses of the 
meaning and significance of such statistics; 

(3) assist State and local educational agencies, including 
State agencies responsible for postsecondary education, in im- 
proving and automating their statistical and data collection ac- 
tivities; 

(4) review and report on educational activities in foreign 
countries: a&4 

(5) conduct a continuing survey of institutions of higher edu- 
cation and local educational agencies to determine the demand 
for, and the availability of, quelified teachers and administra- 
tive personnel, especially in critical areas within education 
which are developing or are likely to develop, and assess the 
extent to which programs administered in theJ jBducati o n jDi v i * " 
w o n are helping to meet the needs identified as a result of 
such continuing survey j and 

(6) access periodically 'the current and projected 
^PPly and demand for elementary and secondary 
school teacher* (including teachers at the pre-school 
level) and early childhood education development 
personnel with particular attention to-- 

(A) long~term and short-term needs tor personnel 
in various subject areas or teaching specialities; 

(B) shortages in particular types of schools 
or communities, and in States or regions; 

(C) the number of minorities entering teaching; 

(D) the proportions of women and minorities 
in educational administration, and the trends over 
time; 

(E) the demographic characteristics, academic 
qualifications, job preparation, experience and 
skills of existing teachers and new entrants in the 
field of education; 

(P) the effect of the introduction of state 
mandated teacher competency teaxe on the demographic 
and educational characteristics of teachers and the 
supply of teachers; and 

(G) the rate at which teachers leave 
teaching, their reasons for leaving, the sources of 
supply for new entrants, and the trends over time^ 
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(cXD There shall be an Advisory Council on Education Statistics 
which shall be composed of 7 members appointed by the Secretary 
and such ex officio members as are listed in subparagraph (2). Not 
more than 4 of the appointed members of the Council may be mem- 
bers of the same political party. 

(2) The ex officio members of the Council shall be — 

eommissbner o f Edu 

(A) ^i? e ffi eg , tef * f ft 6 * Julicmnl lytitttto of Educ a t io n, < —Assistant Secretary 

(VLYtS) the Director of the Census, and y 
(Cj-fBHhe Commissioner of Labor Statistics. 

(3) Appointed members of the Council shall serve for terms of 3 
years, as determined by the Secretary, except that in the case of 
initially appointed members of the Council, they shall serve for 
shorter terms to the extent necessary that the terms of office of not 
more than 3 members expire in the same calendar year. 

(4) The Assistant Secretary shall serve as the non-voting presid- 
ing officer, of the Council. 

(5XA) The Council shall meet at the call of the presiding officer, 
except that it shall meet — 

(i) at least four times during each calendar year; and 

(ii) in addition, whenever three voting members request in 
writing that the presiding officer call a meeting. 

(B) Six members of the Council shall constitute a quorum of the 
Council. 

(6) The provisions of section 448(b) of part D of this title shall not 
apply to the Council established under tnis subsection. 

(7) The Council shall review general policies for the operation of 
the Center and shall be responsible for establishing standards to 
insure that statistics and analyses disseminated by the Center are 
of high quality and are not subject to political influence. 

(d)(1) The Aofliotant Secretary shall, not later than June 1 of each 
year, submit to the Congress an annual report which— 

(A) contains a description of the activities of the Center 
during the then current fiscal year and a projection of its ac- 
tivities during the succeeding fiscal year; 

(B) sets forth estimates of the cost of the projected activities 
for such succeeding fiscal year; 

(C) includes a statistical report on the condition of education 
in the United States during the two preceding fiscal years and 
a projection, for the three succeeding fiscal years, of estimated 
statistics related to education in the United States; and 

(D) clearly sets forth areas of critical need for additional 
qualified education personnel in local education agencies and, 
after discussion and review by the Advisory Council on Educa- 
tion Statistics, identifies priorities within projected areas of 
need, and includes recommendations of the Council with re- 
spect to the most effective manner in which the Nation and 
the Federal Government may address such needs. 

(2) The i Center shall develop and enforce standards designed to 
protect the confidentiality of persons in the collection, reporting 
and publication of data under this section. This subparagraph shall 
not be construed to protect the confidentiality of information about 
institutions, organizations, and agencies receiving grants from or 

having contracts with the Fed era l Government. 

^(eHjuosaeiJa cany out the objectives of the CentenJkfr-AsSET 
ant Secretery-w-mOliomed, either directly ojjnf-*^ 
to carry out the purpoSB^^ (b) ^ for tha { 

purpose the i Assistant^^ make ^anta to 

and contractaJ*rti^ and private instituBoter^gencies, orrani- 

( e 'WWQ The Center is authorized to furnish transcripts or copies 
of tables and ^other statistical records of the Office o f EJuuiliun , 
the Att^Uia C^laij, and ili a . National Institute of Education to 
and 4* make special statistical compilations and surveys for, State 

a J^ ^local officials, public and private organizations, <* individuals. and 

^ / 
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The Center shall provide state and local educational 
agencies opportunities to suggest the development of 
particular compilations of statistics, surveys, and 
^analyses that would assist those educational agencies. 

The Center shall furnish such special statistical compilations and 
surveys as the Committees on Labor and Human Resources and on 
Appropriations of the Senate and the Committees on Education 
and Labor and on Appropriations of the House of Representatives 
may request. Such statistical compilations and surveys, other than 
those carried out pursuant to the preceding sentence, shall be 
made subject to the payment of the actual or estimated cost of such 
work. In the case of nonprofit organizations or agencies, the Aseiefe - 
«rt Secretary may engage in joint statistical projects, the cost of 
which shall be shared equitably as determined bj[ the Assistant 
Secretary: Provided, That the purposes of such projects are other- 
wise authorized bv l a w, . 

TB)^ftiWiuidsreceived in payment for work or services-emnnSr^ 
ated under suEpS5^gwH?h^ a separate ac- 

count which may beuse^ta^Sjrrit^^ costs of such work or 
services, toj^pas^ttppfopnations which ininfflljr-boce^l^or part of 



(2)(A) The Center shall participate with other Federal agencies 
having a need for educational data m forming a consortium for the 
purpose of providing direct joint access with such agencies to all 
educational data received by the Center through automated data 
processing. The Library of Congress, General Accounting Office, 
an<\ the Committees on Labor and Human Resources and Appro- 

Enafr-v^ 0 f the Senate and the Committees on Education and 
abor cdid Appropriations of the House cf Representatives shall, 
for the purposes of this subparagraph, be considered Federal agen- 
cies. 

(B) The Center shall, in accordance with regulations published 
for the purpose of this paragraph, provide all interested parties, in- 
cluding public and private agencies and individuals;, direct access to 
data collected by the Center for purposes of research and acquiring 
statistical information 



^(3) The Commissioner and the National Institute of Educatii 
ar^djrected to cooperate with the Center and make such rgoeffds 
and data^available to the Center as may be necessary to eprfole the 
* Center to caqry cut its functions under this subsectiorjx"*^ 

The fimqunt available for salaries and ^etfpenses of the 
Center shall not exceed $5,000,000 for the fispafyear ending June 
30, 1975, $10,000,000 fbtthe fiscal year epdlng June 30, 1976, and 
$14,000,000 for each of tlfesiiscal years^nding prior to October 1 
1989, 

(2) The amount available fop^aqts and contracts by the Assist 
ant Secretary under subseotlon (e) snkll not exceed $10,000,000 for 
fiscal year 1985, $12^0002)00 for fiscal yw 1986, $14,000,000 for 
fiscal year 1987, $J#flO0,000 for fiscal year 1988 w and $18,000,000 for 
fiscal year l9R r 

(3) Sum^ppropriated for activities and expenses^e^the Center 
which^rfe not limited by paragraph (2) of this subsectibns|hall be 

5priated apart from appropriations which are so limited, as 

Iparate line items. ._. 

(g) CWd) In addition to its other responsibilities, the Nati o nal Center 
for Education Stotiotioo - shall, in - oonoulte ft ion-with tho Department 



(f)(1) Funds to carry 

out this section 
are authorized by 
section 405(g) 
of this Act. 



of Educati o n , collect uniform data from file States on financing of 
elementary and secondary education. Each State receiving funds 
Education under thef Eleiumilary and 3euundary Eduuatiun Act of 1 9 05 ' shall 

Consolidation and cooperate with the Nati o nal Center in this effort. 
Improvement Act ^ Off For purposes of this section, the term's "United" State?' and 

"State" include the District of Columbia and Puerto Rico. 

(20 U.S.C. 1221e-l) Enacted August 21, 1974, P.L. 93-380, sec. 501(a), 88 Stat. 556, 
558; amended April 21, 1976, P.L. 94-273, sec. 12(1), 90 Stat. 378; amended October 
12, 1976, P.L 94-482, sec. 401(c), 90 Stat. 2226; sec. 406, 90 Stat. 2231, 2232; sec. 
501(q), 90 Stat. 2238; amended Nov. 1, 1978, P.L. 95-561, sees. 1201, 1212(a), (c), 
1243(a), 92 Stat. 2333, 2334, 2338, 2341, 2353; see also general reference Oct. 17, 1979, 
P.L. 96-88, sec. 301, 93 Stat. 677; amended Oct. 19, 1984, P.L. 98-511, sees. 702(b), 
704(b), 98 Stat. 2406. 
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GEPA, Section 405(e) 



(1) In addition to the other responsibilities of the Office under 
this section, the Office shall carry out, by grant or cooperative agree- 
ment with a nonprofit educational organization, a National Assess- 
ment of Educational Progre ss which shall have as a primary pur- 
pose the assessment of the performance of children and young 
adults in the basic skills of reading, mathematics, communication, 
and other subjects and skills* Such a National Attessztznt shall— 

(A) collect and report at least once every five years data as- 
sessing the performance of students at various age or grade 
levels in each of the areas of reading, writing, and mathemat- 
ics; 

(B) report periodically data on changes in knowledge and 
skills of such students over a period of time; 

(C) conduct special assessments of other educational areas, 
as the need for additional national information arises; 

(D) include in assessment activities information on special 
groups of individuals; 

(E) provide technical assistance to State educational agen- 
cies and to local educational agencies on the use of National 
Assessment objectives, primarily pertaining to — 

t (0 the basic skills of reading, mathematics, and commu- 
nication, and 

(ii) on making comparisons of such assessments with ihc 
national profile (including special population profiles) and 
change data developed by the National Assessment; and 

(F) with respect to each State which voluntarily participates 
in accordance with paragraph (5), provide a statement of infor- 
mation collected by the National Assessment for each such 
State. 

(2XA) The organization through which the Office carries out the 
National Assessment shall be responsible for overall management of 
the National Assessment Such organization shall delegate author- 
ity to design and supervise the conduct of the National Assessment 
to an Assessment Policy Committee, established by such organiza- 
tion. The Assessment Policy Committee shall be composed of— 

(i) five members appointed by the organization of whom two 
members shall be representatives of business and industry and 
three members shall be representatives of the general public; 
and 

(iO fourteen members appointed by the organization from the 
categories of membership specified in subparagraph (B). 



-5- 



ERLC 




GEPA, Section 405(e) 



(B) Members of the Assessment Policy Committee appointed in 
accordance with subparagraph (AXii) shall be— 
"(V one chief State school officer; 
"(H) two State legislators; 
"(iiV two school district superintendents; 

(iv) one member of a State board of education; 

(v) one member bf a local school board; 

(vi) one Governor of a State; 
(vW four classroom teachers; 



(ix) one secondary school principal 

(C) The Assistant Secretary shall serve as an ex officio member 
of the Assessment Policy Committee. The Assistant Secretary shall 
also appoint a member of the Council to serve as nonvoting member 
of the Assessment Policy Committee. 

(D) Members appointed in accordance with subparagraph (A) (i) 
and (ii) shall be appointed for terms for three years on a staggered 
basis. 

(S) The Assessment Policy Committee established by paragraph 
(2) shall be responsible for the design of the National Assessment, 
including the selection of the learning areas to be assessed, the de- 
velopment and selection of goal statements and assessment items, 
the assessment methodology, the form and content of the sporting 
and dissemination of assessment results, and studies to evaluate 
and improve the form and utilization of the National Assessment. 
The appropriateness of all cognitive, background, and attitude items 
developed as part of the National Assessment shall be the responsi- 
bility of the Assessment Policy Committee. Such items shall be sub- 
ject to review by the Department of Education and the Office of 
Management and Budget for a single period of not more than sixty 
days. 

(V Each learning area assessment shall have goal statements de- 
vised through a national consensus approach, providing for active 
participation of teachers, curriculum specialists, subject matter spe- 
cialists, local school administrators, parents, and members of the 
general public All items selected for use in the assessment shall be 
reviewed to exclude items which might reflect racial, sex, cultural, 
or regional bias. 

(5) Participation in the National Assessment by State and local 
education agencies selected as part of a sample of such agencies 
shall be voluntary. 

(6) The Secretary shall provide for a periodic review of the Na- 
tional Assessment. This review shall provide an opportunity for 
public comment on the conduct and usefulness of the National As- 
sessment and shall result in a report to the Congress, the President, 
and the Nation on the findings and recommendations, if any, of the 
review. The Secretary shall consider the findings and recommenda- 
tions in designing the competition to select the organization through 
which the Office carries out the National Assessment. 
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Part C— Vocational Education and Occupational Information 

Data Systems 

data systems authorized 

Sec. 421. (aXD The Secretary shall develop, within the National 
Center for Education Statistics, a national vocational education 
data reporting and accounting system using uniform definitions. 
The system required by this section shall include information pn 
vocational education— 

(A) students (including information concerning race, sex, and 
handicapping condition), 

(B) programs, 

(C) program completers and leavers, 

(D) placement and followup, 

(E) staff, 

(F) facilities, and 

(G) expenditures in relation to tho principal purposes of this 
Act 

Such information shall include the participation of special popula- 
tions, including women, the disadvantaged, the handicapped, indi- 
viduals of limited English proficiency, and minorities. 

(2) The Secretary snail take such action as may be necessary to 
secure the data required by this section at reasonable cost. The 
Secretary, -in consultation with the Congress, shall determine the 
number and types of vocational education institutions k> be sam- 

Eled, the methodology to be used, group sample sizes, appropriate 
reakdown analyses of such groups, and the frequency with which 
such studies under this section are to be conducted. 

(bXD In maintaining and updating such system, the Secretary 
shall endeavor to the fullest extent feasible to make the system 
compatible with the occupational information system (established 
pursuant to section 422), with the vocational education data system 
authorized under section 161(a) of the Vocational Education Act of 
1963, and with other systems developed or assisted under the Job 
Training Partnership Act and with information collected pursuant 
to the Education of the Handicapped Act. 

(2) Any State receiving assistance under this Act shall cooperate 
with the Secretary in supplying the information required to be sub- 
mitted by the Secretary and shall comply in its reports with the 
vocational education data system developed by the Secretary pursu- 
ant to subsection (a). Each State shall submit the data required to 
carry out this subsection to the Secretary in whatever form the 
Secretary requires. 

(3) The Secretary shall every 2 years update the national voca- 
tional education information and accounting system and prepare 
acquisition plans of data for operating the system. In carrying out 
the requirements under this paragraph, the Secretary shall use sci- 
entific sample surveys for the information required, except that the 
information required with respect to handicapped students shall be 
furnished in accordance with section 423 of this Act. 

(4) The Secretary may conduct special studies on enrollment of 
disadvantaged students in vocational education programs, on the 
participation of handicapped students in vocational education pro- 
grams, and any other similar subjects which the Secretary deems 
appropriate. 

(c) In carrying out the responsibilities imposed by this section, 
the Secretary shall cooperate with the Secretary of Labor in imple- 
menting section 463 of the Job Training Partnership Act to ensure 
that the data system operated under this section is compatible with 
and complementary to other occupational supply and demand in- 
formation systems developed or maintained with Federal assist- 
ance. 

(20 VS.C. 2421) Reenacted October 19, 1984, P.L. 98-524, 98 Stat. 2472. 
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OCCUPATIONAL INFORMATION SYSTEM 

Sec. 422. (a) There is established a National Occupational Infor- 
mation Coordinating Committee which shall consist of the Assist- 
ant Secretary for Vocational and Adult Education, the Commis- 
sioner of the Rehabilitative Services Administration, the Director 
of the Office of Bilingual Education and Minority Language Af- 
fairs, and the Administrator of the National Center for Education 
Statistics of the Department of Education, the Commissioner of 
Labor Statistics and the Assistant Secretary for Employment and 
Training of the Department of Labor, the Undersecretary for Small 
Community and Rural Development of the Department of Agricul- 
ture, the Assistant Secretary for Economic Development of the De- 
partment of Commerce, and the Assistant Secretary of Defense 
(Manpower, Reserve Affairs, and Logistics). The Committee, with 
funds available to it under section 451, shall provide funds, on an 
annual basis, to State occupational information coordinating com- 
mittees and to eligible recipients and shall— 

(1) in the use of program data and employment data, im- 
prove coordination and communication among administrators 
and planners of programs authorized by this Act and by the 
Job Training Partnership Act, employment security agency ad- 
ministrators, research personnel, and personnel of employment 
and training planning and administering agencies (including 
apprenticeship training agencies) at the Federal, State, and 
local levels; 

(2) develop and implement, in cooperation with State and 
local agencies, an occupational information system to meet the 
common occupational information needs of vocational educa- 
tion programs and employment and training programs at the 
national, State, and local levels, which system shall include 
data on r xupational demand and supply based on uniform 
definitions, standardized estimating procedures, and standard- 
ized occupational classifications; 

(3) conduct studies on the effects of technological change on 
new and existing occupational arsas and the required changes 
in knowledge and job skills; and 

(4) assist State occupational information coordinating com- 
mittees established pursuant to subsection (b). 

(b) Each State receiving assistance under this Act shall establish 
a State occupational information coordinating committee composed 
of representatives of the State board, the State employment secu- 
rity agency, the State economic development agency, the State job 
training coordinating council, and the agency administering the vo- 
cational rehabilitation .program. Such committee shall, with funds 
available to it from the National Occupational Information, Coordi- 
nating Committee established pursuant to subsection (a)— 

(A) implement an occupational information system in the 
State which will meet the common needs for the planning for, 
and <ihe operation of, programs of the State board assisted 
under this Act and of the administering agencies under the 
Job Training Partnership Act; and 

(B) use the occupational information system to implement a 
career information delivery system. 

(20 UJ3.C. 2422) Reenacted October 19. 1984, PJL 98-524, 98 Stat 2478. 

INFORMATION BASE FOR VOCATIONAL EDUCATION DATA SYSTEM 

Sec. 423. The Secretary shall assure that adequate information 
on the access to vocational education programs by handicapped sec- 
ondary school students be included in the national vocational edu- 
cation data system, required by section 161 of the Vocational Edu- 
cation Act of 1963 and by this part, for the biennial survey. The 
information base for the biennial survey for the handicappea shall 
be in 4-digit detail as defined in A Classification of Instructional 
Programs published by the National Center for Educational Statis- 
tics. The survey shall include information with respect to total 
handicapped enrollment by program, by type of instructional set- 
ting, ana by type of handicapping condition. 
q (20 UJ3.C. 2428) Rectnactod October 19, 1984, PX. 98-524, 98 Stat 2474. 
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CENTER FOR EDUCATION STATISTICS CCES^ 
Office of Educational Research and Improvement (OERI) 
PUBLICATION CLEARANCE PROCEDURES 

I. Regular Publications 

1. OERI Publication Plan . OERI prepares an annual publication plan, approved 
by the Assistant Secretary, that serves as a device to organize decisions, 
and to record these decisions, on each publication with regard to audience, 
cost and allocation of staff time. Proposed CES publications titles are 
submitted rs part of overall OERI publication proposals for Assistant 
Secretary review in September (or in a memorandum requesting an addition to 
the plan thereafter) . The Office of Director approves or modifies CES 
Division proposals. Information Services (IS) provides oversight, guidance 
and technical assistance for editorial review and production of all OERI 
publications and products . It generates ideas for publications and other 
OERI products, establishes production priorities, tracks development of 
publications, implements editorial review procedures and devises 
dissemination strategies. 

2. Division Review . CES has assigned primary responsibility to Divisions for 
preparation of publications, maintenance of standards and meeting 
timelines. Exact procedures are left up to the Divisions but must 
culminate with Division Director sign-off. At a minimum, sign-off implies 
the data have been checked for accuracy and the draft manuscript meets 
Center standards. 

3. CES Review . CES conducts a Center-wide review as a quality control 
measure. The process is based on the concept of "peer review" commonly 
used in scientific or technical adjudications in statistical agencies, 
other parts of the Government, and academia. Individuals from inside and 
outside the Center participate by reviewing and critiquing drafts, an 
adjudication meeting is held where all issues are discussed and resolved, 
and final revisions are made by the author overseen by the Chief 
Statistician. Principal elements include: peer review, POC review, and 
adjudication of reviewers comments. 

a. Peer Review . Names of at least three reviewers are submitted by 
the Division to the Director, CES. The Director approves the two 
in -house reviewers (one from the Condition of Education Division, 
one from another Division). (The Chief Statistician, or his 
representative, 4 also reviews all publications.) Names of 
external reviewers are submitted by the Director to the 
Assistant Secretary for approval. In practice, the Director, 
CES, approves reviewers for regular CES publications. External 
reviewers are chosen for their expertise and the diversity of 
opinion they bring to the review process; the number of external 
reviewers will vary with the nature, content, complexity and 
intended audiences of publications. 

The approved peer reviewers and the Chief Statistician are given 
copies of the manuscript and are asked to review the technical 
presentation, to determine if the exposition is based on the 
,data, and to ensure the content is set in context. Specifically: 

the Chief Statistician is responsible for technical review and 
adherence to standards; 
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the reviewer from the Condition of Education Division has 
major responsibility for checking consistency vith data in the 
Center's major compendia; 

the reviewer from the other division has major responsibility 
for content and context issues; and, 

the external reviewers are asked to review for subject matter 
and content. 

Written comments are sent to the author prior to the 
adjudication meeting. 

b. POC Review . Copies of the manuscript are sent by the Office of 
the Director to the Assistant Secretary and other pertinent 
department offices (called principa 1 operating components or 
"POCs") for information and written comment. Among the 
Department POCs that may be sent reports are the Offices of the 
Under Secretary; Planning, Budget and Evaluation; 
Intergovernmental Affairs; Management; and all Assistant 
Secretaries. Comments received from POCs are given to the author 
and the Chief Statistician prior to the adjudication meeting. 

c . Adjudication . Timely resolution of peer and POC reviewers 
comments is provided at a meeting scheduled by the Division to 
occur after distribution and critique of the manuscript. The 
Chief Statistician chairs the meeting of peer reviewers at which 
time major points are discussed and adjudicated. The author then 
revises the manuscript and resubmits it to the Chief Statistician 
for sign-off and submission to IS. (Time frames for CES review 
are established by the Divisions and the Chief Statistician when 
the publication plan is developed). 

Respons ibilities of Information Services . Information Services is 
responsible for directing dissemination strategies, the publication 
process and distribution of reports: 

a « Dissemination. IS calls and chairs a dissemination meeting 

involving the author, a representative of the Office of the 
Director, CES, the editor, and the Branch Chief of the Publications 
Branch, to determine what groups (drawn from the OERI mailing lists) 
will receive announcements and publications, whether GPO will sell 
the publication, whether the publication will be made available 
through other sources (e.g. /ERIC, NTIS) , and the number of copies 
to be made available to the Information Office for response to 
requests . 

b « Publication. IS is responsible for editing, copy preparation and 
printing of manuscripts. (In practice, computers, statistical and 
graphics software and laser printers are moving an increasing amount 
of copy preparation to the Center). Note: All substantive changes 
in editing are reviewed and approved by the Chief Statistician. 

c« Distribution. Information copies of publications are sent by IS to 
key offices in the Department a few days prior to external release. 
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II. Special Publications 

Contractor Reports . Work statements calling for contractor reports must 
include provision for revisions of reports and sufficient time must be 
specified between draft and final reports for Division and CES review. 

Clearance procedures are the same as for regular publications except: 

-POC review is ordinarily limited to OERI; 

-Comments from the adjudication meeting are provided to the contractor 
for his consideration; and, 

-The project officer, in addition to determining whether a final report 
satisfys the terms of the contract, recommends, through the Division to 
the Chief Statistician, to the Assistant Secretary whether the 
Government to should publish a final contractor report. 

Grantee Reports . Clearance procedures are the same as for regular 
publications except: 

-The Assistant Secretary approves the external peer reviewers; 

-POC review is ordinarily limited to OERI; and, 

-There is no adjudication meeting. Reviews by the Government are 
submitted to the grantee within 30 days of original submission to the 
Government- Reviewers comments are submitted to the project officer who 
summarizes them and transmits the summary and reviews to the grantee 
through the Director, CES. 

Digest of Education Statistics Clearance procedures are the same as for 
regular publications except: 

-Relevant portions are assigned to the Divisions for review; and, 

-A series of meetings are held vith each Division prior to the 
adjudication meeting. 

Condition of Education Report. This is a Secretarial report, by law. To 
be consistent with Department review requirements for Secretarial reports, 
clearance procedures are the same as for any other Secretarial document. 
However, prior to formal clearance, this report goes through regular CES 
review except that POC review is limited to OERI. 

Ill . Data Tapes 

Data tape announcements and documentation are submitted to and cleared by 
the Chief Statistician and the ADP Coordinator. 

IV. General Note . At all stages of the publication process, except for the 
Condition of Education report, the Director, CES, is the final judge of the 
content of CES publications. 
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Office of Educational 
Research and Improvement 

U& Department cf Education 



EXTERNAL ADVICE AND PEER REVIEW 
in the 

OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT 
Effective August 18, 1986 



POLICY STATEMENT: 

It is the policy of the Office of Educational Research and Improvement 
to seek advice from outside experts on all major program decisions. For 
this reason, principles of peer review permeate all aspects of important 
OERI decisions. These include but are not limited to decisions 
pertinent to program planning and development, the review, evaluation 
and redirection of OERI systems and programs, appraisals of reports and 
publications, and other aspects or dimensions of quality control 
throughout the organization. 

Peer review is a concept or principle that gives rise to a wide array of 
practices. It is not a single uniform procedure. In its simplest and 
most general form, the use of peer review involves obtaining advice 
about a decision. 



For a federal agency such as OERI, peer review represents and protects 
that which all government officials vow to serve— namely, the public 
interest. OERI uses peer review to improve the quality, utility, and 
worth of all its research, data gathering, and dissemination activities. 
It is also used to enhance OERI's accountability, to introduce into OERI 
a broader range of viewpoints and ideas, and it is a valuable means of 
updating knowledge of relevant developments and findings from outside 
sources. 

Peer review in OERI means obtaining advice so that the decisions made by 
OERI are informed by sound advice from multiple and often competing 
perspectives. A number of studies of the peer review process in other 
fields and agencies indicate that much of its benefit derives from 
informed disagreement. Indeed, the interactions associated with such 
disagreement are powerful self-correcting mechanisms for scientific and 
other forms of inquiry. 

For these reasons, so long as peer review yields an array of 
well-founded advice, it need not yield group consensus. To be sure, 
there are times when it is desirable to seek agreement from reviewers 
functioning as a group. This is especially true in determining which of 
many proposals for a major project are so deficient that they ought not 
remain within the competitive range. Yet even where formal evaluations 
complete with point scores are provided by panels of reviewers, totals 
and averages can be calculated even if reviewers do not agree with each 
other. 
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A truly effective peer review system should yield intellectual ferment, 
creative disagreement,. and testing of conclusions and assumptions in the 
"marketplace of ideas" which, in turn, will be enriched and enlarged by 
the process. Furthermore, a truly effective system will foster 
diversity, originality, and innovation in thinking and not simply 
project current conventional wisdom into the future. In so doing, it 
will guard against dated assumptions, provoke new ideas, and seek 
unconventional approaches and new strategies. 

Consensus will be welcomed where it is authentic and critical, but 
agreement will not be feigned where there is none. More often than not, 
peer review in OERI should produce informed debate and creative 
disagreement in disciplined forms that deepen knowledge, widen critical 
understanding, and enhance practice. 

Central to the process of peer review in OERI is the conviction that 
reviewers should be drawn from the varied constituencies and audiences 
of OERI, as well as from the academic community. Individual advisors 
and reviewers must have no direct stake in the outcome of the decision. 
That is, advice is solicited from persons for whom there is no conflict 
of interest. This includes both an actual conflict of interest as well 
as the appearance of one. Furthermore, peer reviewers must have no 
other constraints on their ability to supply their best individual 
judgments. 

The combination or mix of reviewers will vary according to the specific 
requirements of the situation. For example, it is often desirable for 
OERI outside reviewers to include parents, teachers, principals, and 
other school practitioners. In addition, state and local officials and 
professional staff members, journalists and social critics, business and 
labor leaders, authors and analysts, and librarians and library users 
are all potential members of peer review panels. 

In short, a peer reviewer's credentials may come from academia as well 
as life experience. What unites this diverse group from different walks 
of life is their interest in and commitment to education and their 
ability to make cogent well-informed contributions to the OERI peer 
review process based on their perspectives, knowledge, and experiences. 
Their advice is of special value to OERI. 
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EXTERNAL ADVICE AND PEER REVIEW: 
OERI IMPLEMENTATION PROCEDURES 
August 18, 1986 



SCOPE: 

In accordance with its external advice and peer review policy, the Office of 
Educational Research and Improvement (OERI) seeks the advice of outside 
experts about all major program activities, including: 

PROGRAM PLANNING 

PROCUREMENTS 

EVALUATION OF OERI SUPPORTED ACTIVITIES 
PUBLICATIONS AND PRODUCTS* 

It is the intention of OERI that decisions in the categories cited above be 
informed by external advice and peer review. In no instance, however, does 
implementation of OERTs external advice and peer review policy absolve the 
Secretary of Education, the Assistant Secretary, or any OERI official of 
responsibility for activities undertaken or decisions made by OERI. 
Reviewers* assist, but they do not decide; final decisions are made by OERI. 



RESPONSIBILITIES OF OERI PROGRAM DIRECTORS : 
Program Directors, working with professional staff, have primary 
responsibility for ensuring that the work of OERI adheres to the highest 
standards of excellence and intellectual rigor. They are held accountable 
for ensuring that the principles of peer review permeate all of OERI 1 s 
activities. OERI Program Directors have considerable flexibility in planning 
for external advice and peer review and are the first line of decision in 
determining whether plans and procedures for obtaining peer review and advice 
from outside experts are soundly conceived, appropriate to the tasks, 
conducted with integrity, and well documented. 

All OERI activities must reflect OERI 1 s commitment to peer review and 
external advice. Plans for OERI program initiatives must address the ways in 
which the advice and counsel of outside experts will be sought. Program 



*Note: See "OERI Publications and Products: External Advice and Peer Review 
Implementation Procedures". 

For purposes herein, the terms reviewer and advisor may be used 
interchangeably to refer to outside experts, e.g., researchers, 
statisticians, methodologists, policymakers, parents, education 
practitioners, etc., from whom OERI seeks advice. 
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Directors are to submit these plans through the Deputy Assistant Secretary 
for Policy and Plannino to the Assistant Secretary for review and approval. 
Program Directors also must incorporate into their budget plans for 
particular activities, the costs associated with obtaining external review 
and advice. This is especially important where major decisions-making 
processes may warrant relatively high-cost review procedures. 



RESPONSIBILITIES OF OERI PROFESSIONAL STAFF; 

The professional involvement of OERI program staff is critical to the 
successful implementation of the external advice and peer review policy. 
Specific responsibilities of program staff under this OERI policy are 
included in their individual performance agreements and plans. 

In general, program staff are responsible for the management and direction of 
the review process throughout its duration as well as for the provision of 
logistical support. They are responsible for determining the nature of the 
advice or review required for program plans, issues, initiatives, 
publications, etc., and for framing the issues about which advice or review 
is sought. Program staff also will identify potential reviewers, develop 
review strategies, and prepare and forward required decision packages through 
the Program Director to the Assistant Secretary for review and approval. It 
is expected that program staff will brief and orient reviewers, synthesize 
reviewers' comments, ensure the appropriate and timely involvement of other 
OERI staff and officials, and share their own professional opinions and 
judgments. Professional staff also will work closely with the Department's 
Grants and Contracts service to ensure productive negotiations and timely 
awards. 



REVIEW STRATEGIES; 

Based upon the stated purposes of a particular review or the rationale for 
seeking the advice of outside experts, OERI Program Directors and their 
professional staffs have a range of possible strategies from which to choose. 
These include, but are not limited to: (1) intensive technical reviews of a 
single proposal by individual reviewers; (2) review and discussion of several 
proposals by reviewers convened in a single location or communicating via 
telephone conferencing; (3) a two tiered review process involving a technical 
review at the first level and an assessment of policy, program, or practice 
implications at the second level or vice versa; or (4) provision of advice on 
program plans and direction by a study panel established for that purpose. 

It should be noted that the processes used to obtain external advice and peer 
review will not always include face-to-face meetings of reviewers, 
traditionally referred to as a panel process. Research has indicated that 
groups not meeting in person, but whose members are contacted by mail and 
phone, can be as effective as panels. Indeed, these nominal groups, as they 
are called, possess certain advantages. For example, they avoid the 
possibility of having an overpowering personality dominate panel meetings, 
thereby inhibiting others in expressing dissent. In addition, mail reviews 
are more economical. The latter is particularly important during periods of 
scarce resources. 
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REVIEWER SELECTION: 

The mix of reviewers will vary according to the specific requirements of the 
situation. For example, appraising the competing merits of alternate sample 
designs for a statistics-gathering project or gauging the validity of a 
technical analysis of a large scale data set are tasks for which OERI would 
best be served by reviewers with specialized expert knowledqe. But selecting 
those issues, of, say, the preparation and certification of school administrators 
to which OERI might most appropriately apply some of its research resources 
is a task wherein the views of practitioners and policymakers are particularly 
germane. As another example, when evaluating which of the findings of prior 
research in early learning might be of greatest public interest and value, 
parents as well as researchers and practitioners certainly should be enlisted 
in the peer review process. 

Reviewers may be selected in part from among knowledgeable employees of the 
federal government, provided that they have no conflicts of interest or 
constraints on their ability to supply their best professional judgments. 
Selection of federal employees as reviewers should occur in the following 
descending order of preference: 

(1) employees of agencies other than the U.S. Department of 
Education; 

(2) employees of Department of Education units other than OERI; 

(3) staff of an OERI Program other than the Program conducting the 
review or soliciting advice; 

(4) staff of other Divisions within the OERI Program conducting the 
review or soliciting advice. 

In selecting individuals to provide external advice, conflict of interest, 
either actual or apparent, must be avoided. There must be no doubt as to 
whether an individual has financial or other interests in the outcome that 
would raise questions about his or her objectivity. In general, a conflict 
of interest situation exists when any of the following have a financial or 
other interest in the outcome: 

(1) the individual being considered, his or her spouse, minor 
child, or partner; 

(2) a profit or nonprofit organization in which the individual 
being considered is serving as an officer, director, trustee, 
partner, or employee; 

(3) any person or organization with whom the individual being 
considered is negotiating or has an arrangement concerning 
prospective employment. 

Inquiries from OERI professional staff concerning conflict of interest 
guidelines as they pertain to the selection of external reviewers should be 
directed to OERI's Planning Coordinator in the Office of the Deputy Assistant 
Secretary for Policy and Planning. 

Individual advisors or reviewers can be suggested or nominated by any OERI 
professional staff member, any other employee of the Department of Education, 
any other Federal employee, or members of the general public. These 
nominations may be received by any OERI professional staff member. They then 



ERIC 



-5- 



OERI CENTRAL FILE: 

The OERI Central File of external reviewers should be consulted by Program 
Directors and professional staff whenever they need to develop a slate of 
reviewers. Information Services (IS) will manage and maintain the File. The 
OERI Central File will be an automated system and should contain relevant and 
appropriate background information useful to Program Directors and 
professional staff in developing their lists of potential reviewers. 

When approval to use an individual as a reviewer is received, the cognizant 
Program Director must submit to IS a completed External Reviewer Information 
Sheet and a resume. IS will review the form for accuracy and completeness 
and enter the information into the OERI Central File. IS also is responsible 
for annotating the File as to the instance and manner in which an individual 
was used for external advice or peer review. In addition, IS will conduct an 
annual audit of the File to update information about each reviewer. A 
printout from the File will be suomitted for review and revision to the 
professional staff member who submitted the original in f ormation or to the 
appropriate Program Director. 

Any changes in the status of an individual, e.g., change in employment, that 
might alter the initial judgement that an individual is to be included in the 
File is to be brought to the attention of the Program Director. The Program 
Director then will recommend an appropriate course of action to the Deputy 
Assistant Secretary for Policy and Planning. 



REVIEWER COMPENSATION: 

Those who serve as peer reviewers or provide advice to OERI perform an act of 
public service involving a contribution of time and expertise. In order to 
acknowledge this valuable service, a letter of appreciation is sent by the 
Assistant Secretary to each reviewer following completion of the proceedings. 
This letter is prepared by program staff. Two other forms of recognition are 
provided to acknowledge the individual's public service. First, OERI will 
distribute nationally, on a periodic basis, a list that publico,lly recognizes 
the contributions of those who have served as reviewers. In addition a 
certificate of appreciation is awarded to these individuals. Information 
Services is responsible for coordinating these activities. In some 
instances, Program Directors may recommend that the Assistant Secretary 
approve the granting of honoraria for services rendered. These occasions 
must be consistent with Department-wide policy. They must be justified and 
submitted in writing to the Assistant Secretary for consideration and 
approval. Program Directors are expected to take whatever action is 
appropriate to generate the sense of obligation and public service necessary 
for causing individuals to be willing to serve as potential reviewers. 

Within the limits of its resources and consistent with Department of 
Education policies and prudent fiscal management, OERI will reimburse 
reviewers for necessary and reasonable expenses incurred for their iervice. 
However, Program Directors must minimize costs whenever possible by using 
economical procedures such as reviews by mail and conference telephone calls 
rather than meetings which involve reviewer travel. Our concern for quality 
is rivalled only by our insistence on prudence and parsimony in the 
deployment of taxpayers 1 money. 



9 

ERIC 



64 



